It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.

What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.

Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.

The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.

▲

therobots927 5 hours ago | parent [-]

A literal bird brain would outperform an LLM on most spatial reasoning tasks.

Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator?

I think it’s pretty clear that this is a dead end.

▲

ACCount37 5 hours ago | parent | next [-]

Do birds expose enough of their cognition through birdsong?

Do birds expose locomotion-relevant functions specifically through birdsong?

Do we have enough birdsong data available to start solving the inverse problem?

If "yes" on all, then we might be able to do it.

I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".

The last open question is: is there enough spatial reasoning reflected in the language data we have?

It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.

We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.

	▲	3 hours ago \| parent \| next [-]
		[deleted]
	▲	therobots927 2 hours ago \| parent \| prev [-]
		I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities. I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily” I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone. As far as ARC-AGI goes it looks like last years models were scoring <5% against their v2 benchmark: https://arxiv.org/pdf/2505.11831 Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.

▲

jpatten 5 hours ago | parent | prev [-]

Out of curiosity I gave Fable (on max effort) a CAD task yesterday, which was to design a space efficient carrying case for a set of fasteners in my repair kit for work. It used CadQuery to generate a STEP file. The result was pretty much exactly what I wanted, without needing any manual edits. I did go back and forth with it on the design, but was really impressed with the result. Without prompting it included nice touches like ribs on the bottom of the lid to stop fasteners from migrating to adjacent compartments, and the right tolerance for the fit between the case and the lid. This is a dramatic improvement from Opus 4.8.

	▲	therobots927 2 hours ago \| parent [-]
		Well the thing about CAD files is that through reinforcement learning you can basically ask the AI to generate the CAD file an arbitrary item - say it’s a rabbit. It might have examples of this already in its training set and it’s essentially a similarity lookup - but for sake of argument assume we are giving it examples at the edge of the distribution (the whole point of RL). It guesses and you render the file. You pass that image to another AI (not being trained) and ask it if it resembles the description you gave the AI in training. If it does, you have a positive example. If it doesn’t, negative. In that way you can essentially apply transfer learning from the image recognition functionality to the description -> CAD functionality. But is that actually spatial reasoning? Or is it effectively image generation? Because there’s a difference. Spatial reasoning implies that you could drop it in a video game, give it rules, and let it run. And it would play the game well. Like a flight simulator. That would be true spatial reasoning because spatial reasoning is not just identifying objects but understanding how they interact with one another in a highly quantitative way.