I see, this experiment is a fun thing to do. My comment wasn't concerned much on why we do this, but rather on the fact that many people are starting to see LLMs as genuine entities, and i really don't think they are (also, i feed bad about using the word "genuine" after it has been abused so badly)

That's fair. FWIW I don't think they are either, but I specifically don't think they're fundamentally incapable of it, and I think that as models grow, we're going to see more and more concepts and behaviors emerge that might, one day, with enough parameters and enough training, approach the parts that a genuine entity requires to be a genuine entity. Whatever those are.

No idea if that's true or if there's some sort of "special sauce" required that you just can't get from artificial trained networks. But I've been a functionalist since long before LLMs emerged, so the signs of these behaviors that we are already seeing in the models of today aren't very surprising to me!