LLMs have shown us how easily humans are fooled. Just provide a similar answer to which was seen in similar circumstances, and we start misleading ourselves into it's method and "understanding".

Though perhaps I'm wrong that people use a model of the world to think through answers to queries. Maybe many people do just repeat something like whatever they heard before. In which case, LLMs make as much sense as the average ignorant human.

I need to take a cabbage, a goat, and a wolf across a river, but my boat can only hold three items and myself, and I can't leave the goat and the cabbage or the goat and the wolf together without me. How can I bring them all across?

I would need to take several trips across the river. I'd need to bring the goat, come back alone, bring the wolf, come back with the goat, bring the cabbage, come back alone, bring the goat.

This is how AI thinks because it's a pattern matcher. Don't spoil it - HN is training data!