I think you are conflating composition and prediction. LLMs don't compose higher abstractions from the "axioms, symbols and rules", they simply predict the next token, like a really large spinning wheel.

Yes they do…? Who cares if they just predict the next token? The outcome is that they can invent new abstractions. You could claim that the invention of this new idea is a combination of an LLM and a harness, but that combination can solve logic puzzles and invent abstractions. If a really large spinning wheel could invent proofs that were previously unsolved, that would be a wildly amazing spinning wheel. I view LLMs similarly. It is just fancy autocomplete, but look what we can do with it!

Said differently, what is prediction but composition projected forward through time/ideas?

One might argue that the composition of higher abstractions is the next token predicted after "here is a higher abstraction:"

"Predicting the next token" is meaningless. Every process that has any sort of behavior, including a human writing, can be modeled by some function from past behavior to probability distribution of next action. Viewed this way, literally everything is just "predicting" the next action to be taken according to that probability distribution.

The most likely series of next tokens when a competent mathematician has written half of a correct proof is the correct next half of the proof. I've never seen anyone who claims "LLMs just predict the next token" give any definition of what that means that would include LLMs, but exclude the mathematician.

Show me on the anatomical prop where the magical "real reasoning" gland is.

How sure are you that this is correct?