Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it.

The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.

LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.

People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.

Reasoning is a term that's like the term intelligence in that it's highly complex and much less well defined than we'd like it to be, especially as we're trying to compare algorithms to human behaviors.

To begin to explore this we have to imagine a reasoning oracle, that when presented with a question that can be reasoned will immediately present a correct answer. Immediately from this definition we can see that humans are not perfect reasoning oracles. Also with definition of reasoning, any agent with generalized capabilities would quickly reach some kind of singularity or solution to the universe as much as is possible within this universe.

We've established that we are imperfect reasoners, the next question ends up being how do you measure reasoning capabilities? This comes with a second question of when does following an algorithm become reasoning? Animals for example have a large amount of instinctual reasoning as an inherit part of their architecture that self trains when presented with the correct environment. What does the gradient between low level reasoning and high level reasoning look like?

Coming back to humans, we know of countless example of where human reasoning was right for the wrong reasons. We also have seen many different paths to the same solution. For the egotistical it would be easy to say "well if you didn't do it they way I did, you are wrong", but that's not very scientific. Defining the topology of the problem is a more scientific way of looking at it, but quickly runs into P=NP and Kolmogorov complexity issues.

The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning. Therefor I'm highly resistant to consider human reasoning "real" reasoning versus any other possible path to reasoning. Human reasoning is simply what we are comfortable with. This also leads me to consider the worrying possibility there are far more efficient and faster means of reasoning that I as a human will be very uncomfortable with.

Aren't humans also classifiers? Where, precisely, is the dividing line between a sufficiently large model and an intelligence?