Is it really?

I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.

Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.

When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.

It’s all pattern matching trained on a reward function.

There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.

Maybe, but I suspect that a lot of the actual things that produce the algorithm, proof, etc are not actually explicit reasoning. Q: "Why did you try that vs some other thing at this point?" A:"seemed right to do"/"kinda looks like other stuff I've seen before"/"first thing that came to mind" etc.

Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not.

Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.

Of course, but it is rare.

Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either.

I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.