You're right that there is no purely mathematical argument; it's almost non-sensical to claim such.

Instead you can simply make the rather reasonable observation that LLMs are a product of their training distribution, which only contains partial coverage of all possible observable states of the world. Some highly regular observable states are thus likely missing, but an embodied agent (like a human) would be able to correctly reason about these states due to other sensory modes that observe these states. It's very clear that the number of possible states far exceeds any text describing those states (the number of states transcribed to text is vanishingly small in comparison, but they are locally coherent and quite regular when observed through other sensory modes like vision). That's the closest you can make to a mathematical argument.

As a result one common failure case for an LLM might involve describing a specific physical environment and associated set of actions that are in an area of low (or no) probability mass in the training data and asking the LLM to reason about it. Here's an example where I asked ChatGPT the following:

  imagine i walked into a room through a doorway, then turned around and walked right back out of the room. from the perspective of a person in the room facing the door, which side would my right arm be? please explain.

  Let’s break this down step by step.

  1. *You walk into the room through the doorway*:

   * As you're entering, the person in the room is facing the door. So, you're entering from the **outside**.
   * Your **right arm** will be on the **right side** of your body, just as it is normally when you're standing.

  2. *You turn around and walk right back out*:

   * When you turn around to walk out, you’ll be facing **away** from the room, looking back toward the doorway.
   * Since you're walking **back out**, the person inside the room is still facing the door, while you are walking away from them, so your **right arm** is still on the right side of your body.

  So, from the perspective of the person in the room, as you’re exiting, your *right arm* will still be on their *left side*. That’s because they’re facing the door, and your right side is to their left when you’re turning around and leaving.

  To visualize:

  * Imagine a door in front of them.
  * You walk through and turn around, so you're now facing away from them.
  * Your right arm is on your right side, but to them, it's on their left side since you’re now moving in the opposite direction.