> We expect our agents, being flesh and blood humans, to have persistence, to socially respond indefinitely into the future due to our interactions, and to have some give-and-take in response to that.

I fundamentally disagree. I don't go around treating people respectfully (as opposed to, kicking them or shooting them) because I fear consequences, or I expect some future profit ("iterated game"), or because of God's vengeance, or anything transactional.

I do it because it's the right thing to do. It's inside of me, how I'm built and/or brought up. And if you want "moral" justifications (argued by extremely smart philosophers over literally millennia) you can start with Kant's moral/categorical imperative, Gold/Silver rules, Aristotle's virtue (from Nicomachean Ethics) to name a few.

This sounds like you have not thought a lot about how you define those words you use "the right thing to do".

There are indeed other paths to behavior that other people will find desirable besides transactions or punishment/reward. The other main one is empathy. "mirror neurons" to use a term I find kind of ridiculous but it's used by people who want to talk about the process. The thing that humans and some number of other animals do where they empathize with something they merely observe happening to something else.

But aside from that, this is missing the actual essense of the idea to pick on some language that doesn't actually invalidate the idea they were trying to express.

How does a spreadsheet decide that something is "the right thing to do"? Has it ever been hungry? Has it ever felt bad that another kid didn't want to play with it? Has it ever ignored someone else and then reconsidered that later and felt bad that they made someone else feel bad?

LLMs are mp3 players connected up to weighted random number generators. When an mp3 player says "Hello neighbor!" it's not a greeting, even though it sounds just like a human and even happened to the words in a reasonable context, ie triggered by a camera that saw you approaching. It did not say hello because it wishes to reinforce a social tie with you because it likes the feeling of having a friend.