Ignoring its negative connotation, it's more likely to be a highly advanced "stochastic parrot".

> "You don't do that without some kind of working internal model of mathematics."

This is speculation at best. Models are black boxes, even to those who make them. We can't discern a "meaningful internal representation" in a model, anymore than a human brain.

> "There is just no way you can get to the right answer by spouting out plausible-sounding sentence completions without understanding what they mean."

You've just anthropomorphised a stochastic machine, and this behaviour is far more concerning, because it implies we're special, and we're not. We're just highly advanced "stochastic parrots" with a game loop.

> This is speculation at best. Models are black boxes, even to those who make them. We can't discern a "meaningful internal representation" in a model, anymore than a human brain.

They are not pure black boxes. They are too complex to decipher, but it doesn't mean we can't look at activations and get some very high level idea of what is going on.

For world models specifically, the paper that first demonstrated that LLM has some kind of a world model corresponding to the task it is trained on came out in 2023: https://www.neelnanda.io/mechanistic-interpretability/othell.... Now you might argue that this doesn't prove anything about generic LLMs, and that is true. But I would argue that, given this result, and given what LLMs are capable of doing, assuming that they have some kind of world model (even if it's drastically simplified and even outright wrong around the edges) should be the default at this point, and people arguing that they definitely don't have anything like that should present concrete evidence ot that effect.

> We're just highly advanced "stochastic parrots" with a game loop.

If that is your assertion, then what's the point of even talking about "stochastic parrots" at all? By this definition, _everything_ is that, so it ceases to be a meaningful distinction.