> When researchers attempt(opens a new tab) to recover [something like] a coherent computational representation of an Othello game board they instead find [bags of heuristics]

Humans don't exactly have a full representation of board space in their head either. Notably, chess masters and amateurs can memorize completely random board positions as well as the other. I'd think neither could memorize 64 chess pieces in random positions on a board.

That's not what "coherent computational representation" means in this context. It means being able to reliably apply the rules of Othello / chess / etc to the current state of the board. Any competent amateur can do this without studying thousands of board positions - in fact you can do it just from the written rules, without ever having seen a game - they have a causal, non-heuristic understanding of the rules. LLMs have much more trouble: they don't learn how knights move, they learn how white knights move when they're in position d5, then in position g4, etc etc, a "bag of heuristics."

Notably this is also true for MuZero, though at that scale the heuristics become "dense" enough that an apparent causal understanding seems to emerge. But it is quite brittle: my favorite example involves the arcade game Breakout, where MuZero can attain superhuman performance on Level 1 and still be unable to do Level 2. Healthy human children are not like this - they figure out "the trick" in Level 1 and quickly generalize.

For whatever its worth, I bet the chess master would be able to instantly identify that it is a random/invalid board position, aka an invalid world state. I think the experiment you are alluding to gave both groups a very limited amount of time to look at the board. Given enough time, both groups would definitely be able to memorize 64 pieces on a board.