Yes it's about them not being human.

No it's not about humans being irrationally competitive. Human limitations on conversation length, bandwidth, research speed, etc are severe, creating a prisoner's dilemma around open-mindedness that usually makes it an unstable strategy. At any point, your conversation partner can choose to abuse the fact that confident lies take 1x effort to tell and 10x-100x effort to debunk -- unless you are both in a context that actually discourages this behavior, which is rare. Closed-mindedness is a Nash Equilibrium.

Instead, LLMs can be more persuasive due to economics. An LLM doesn't have to worry that it is wasting its resources trying to logic someone out of a position that they didn't logic themselves into, or worse, dumping the effort into a conversation with a bad-faith actor intent on exploiting the misinformation asymmetry. The resource allocation question was answered before it was even invoked, by the person paying to run it. The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.

> The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.

I suppose that's a fair point as well. Though, if I'm arguing with a human - and they pull up ChatGPT to make their points and do their arguing for them, I would consider that bad-faith. Even if it might be the same exact dialog as if I pulled out my phone and discussed it with AI, without of the human middle-manning. Maybe I'm just particularly sensitive, but for me, there's something about my argument being with a real human that makes it much more emotionally charged, and prompts my mind to close. I'm aware of this and try to resist, but it's I think very natural