The vendors like to toss around terms like “thinking” or “reasoning” to encourage prospective buyers to anthropomorphize models. These challenges are a visceral reminder that none of those marketing claims accurately describe was LLMs do: they’ll happily return things even the worst human illustrator would never hand in and make errors showing that there’s no model of the world behind anything they do.

That doesn’t mean there are no ways to use them productively but rather that you should keep in mind that the same model will happily give you code or a decision with the same level of error unless you have carefully setup a QA regimen to prevent that.