Gonçalo Esteves / capable under pressure

The Vault / Open questions

Open question

What sensors does a model need in order to understand the physical world?

Why it matters to him: he owns the test rig. A studio, students, and a body of martial knowledge is the one place this question can be answered rather than speculated about — and it is the only question on either tape where his accumulated advantage is the input. Tagged NAME ON I

status · captured added 30 Aug 2026 asked ×4 from Moonshots #266 trainingspace

Why it matters to him: he owns the test rig. A studio, students, and a body of martial knowledge is the one place this question can be answered rather than speculated about — and it is the only question on either tape where his accumulated advantage is the input. Tagged NAME ON IT.

Where it has come up

asked_count: 4 — he returned to it four separate times in one 74-minute tape, restarting the thought twice on the record ("Take three… Take four"). A question asked four times is not a note, it is a direction. This is the strongest single signal in the archive so far.

  • Moonshots #266 @~9:00 — the panel calls LLMs "librarians that don't get out." He answers from the mat: "There's books about the physics of karate, the physics of Krav Maga… If you read all of that stuff, you will not learn martial arts. That's it."
  • Moonshots #266 @~10:30 — "What are the sensors that we can put on LLMs that increases their capability to understand the physical world?"
  • Moonshots #266 @~11:12 — is it an extension of the human body reporting back to a model, or a human using a model with sensors? His answer: "it could be both."
  • Moonshots #266 @~1:07 — the fourth take: a multimodal model, video + pictures + heart rate, that could train people in self-defence.

What is known so far

The term he was reaching for is embodied AI. His framing splits cleanly in two and the split is his own: (a) what does a model need to sense, and (b) where do the sensors sit.

What would answer it

Not a literature review — a measurement. One technique, one student, one sensor set, and whether the model's judgement matches his own. He is the ground truth, which is the part nobody else has.

Related

  • Embodied AI
  • Existing board project: Do Not Die On The Mat — the 06-28 analysis found this question is that project's R&D horizon, not a new company.
Reading rule. Anything marked unverified or on-air is a claim made on the podcast and has not been checked against a primary source yet. It stays marked until it has. Sourced numbers link to where they came from.