> Heck, just for fun I asked a reasonably smart LLM to ...

LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.

> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...

Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.

Nothing more.

See also anthropomorphism[0].

> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.

This still falls under the purvey of statistical token generation. To wit, given enough variations of:

  bc -e '1 + 2'
  bc -e '41 + 1'
  ...

It is pattern recognition, a task in which ANNs[1] excel.

0 - https://en.wikipedia.org/wiki/Anthropomorphism

1 - https://en.wikipedia.org/wiki/Neural_network_(machine_learni...

By this logic a human is only $130-$160 worth of Oxygen, Carbon, Nitrogen and some trace elements. Perhaps structure sometimes makes things that are more valuable than their inputs?

That said, this is also inaccurate at a technical level.LLM's are very capable of doing math and they ARE calculating internally. Most of what they do is calculation, not storage. It's just not done in a way that it's trivial to explain here.

It's described in some detail below, though it's a bit dense.

https://www.lesswrong.com/posts/E7z89FKLsHk5DkmDL/language-m...

>LLMs are neither smart nor stupid.

by that reasoning then neither are there smart or stupid designs, questions, answers, or any of the millions of things that were described as smart or stupid, that did not possess any brain to actually be smart or stupid long before LLMs showed up.

The analogical process implied in many common English usages means that describing an LLM as smart or stupid is perfectly reasonable.

I'm not anthropomorphizing anything, I literally said that the training data for the formulas and equations is baked into it. It only "knows" things because a crawler and scraper acquired the information from an existing written source. In just about the same way that information is baked into a printed encyclopedia.

During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously. It encompasses virtually everything. It is totally meaningless. So to say "nothing more" is effectively also a tautology.

This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?

It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.

Look at the proof of this: https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.

If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.

> They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.

You haven't demonstrated why this matters.

> Nothing more.

Are you contending that complex systems cannot be more than the sum of their parts?

A market is nothing more than offers and counter offers.

A ant colony is nothing more than scent trails.

All life on earth is nothing more than reproduction with variation.

> This still falls under the purvey of statistical token generation.

Stating the mechanism does nothing to provide insight into capability. For instance: a nuclear power plant boils water by using fuel rods for heat. What does that tell us about the capability of nuclear power?

> This is not "doing" or "understanding" math.

Asserting something purely by stating it does not prove anything but that you intuitively believe it to be true.