You would be correct to be amazed if someone was arguing:

"Look! It made mistakes, therefore it's definitely not reasoning!"

That's certainly not what I'm saying, anyway. I was responding to the argument actually being made by many here, which is:

"Look! It plays pretty poorly, but not totally crap, and it wasn't trained for playing just-above-poor chess, therefore, it understands chess and definitely is reasoning!"

I find this - and much of the surrounding discussion - to be quite an amazing display of people's biases, myself. People want to believe LLMs are reasoning, and so we're treated to these merry-go-round "investigations".