This just confuses me so much. They know the definition of what "self" means, is it just broken because this is the first time they can interact with another seemingly complex system without feeling exploited or manipulated? Have they never cooperated with another person with a mutual goal? Not that LLMs would ever be on that level.

Here is my hobbyist take: There are certain types of personalities that do not form deep connections. They "use" people to suit their needs. Machiavellians and Narcissists come to mind. Especially for them, AI must be godsend. It tells them they are right, it listens, it admires. It‘s basically a perfect version of their perceived self.