We care whether it's reasoning or not because the alternative is that it's guessing, rather than reasoning, and when guessing is measured on benchmarks that are supposed to measure reasoning, the results are likely to be misleading.

Why do we care if the benchmark results are misleading? The reason we have benchmarks in machine learning is that we can use the results on the benchmarks to predict the performance of a system in uncontrolled conditions, i.e. "in the real world". If the benchmarks don't measure what we think they measure then they can't be used to make that kind of prediction. If that's the case then we really have no idea how good or bad a system really is. Seen another way, if a benchmark is not measuring what we think it measures, all we learn from a system passing the benchmark is that the system passes the benchmark.

Still, what do you care if it gets you the right answer? The question is, exactly, how do you know it's really getting you the right answer? Maybe you can tell when you know the answer, but what about answers you genuinely don't know? And how often does it get you the wrong answer but you don't realise? You can't realistically test an AI system by interacting with it as thoroughly and as rigorously as you can with... a benchmark.

That's why we care about having accurate benchmarks that measure what they're supposed to be measuring.

P.S. Another issue of course is that guessing is limited while reasoning is... less limited. We care about reasoning because we ideally want to have systems that are better than the best guessing machine.

“ In many reasoning-heavy benchmarks, o1 rivals the performance of human experts. Recent frontier models1 do so well on MATH2 and GSM8K that these benchmarks are no longer effective at differentiating models. We evaluated math performance on AIME, an exam designed to challenge the brightest high school math students in America.”

https://openai.com/index/learning-to-reason-with-llms/

The benchmark was so saturated that they didn’t even bother running it on the newer models.

Which is interesting because it shows the rapid progress LLMs are making.

I’m also making a bigger claim - you can’t get gpt-5 thinking to make a mistake in undergraduate level maths. At least it would be comparable in performance to a good student.

Sure I didn’t say it was perfect. But questioning the essence of the article.

Example?

Make up one yourself and try it?

"if you don't let chatgpt search the internet or use python code, it doesn't count..."

look at those goalposts go!