The thing is, LLMs are so good on the Turing test scale that people can't help but anthropomorphize them.

I find it useful to think of them like really detailed adventure games like Zork where you have to find the right phrasing.

"Pick up the thing", "grab the thing", "take the thing", etc.

AI Dungeon 2 was peak AI.