It's interesting though. You can have a "relationship" with an influencer. You act as if you knew them and as if they were your friends, you imitate them in what they say and do, talk to them in your mind, follow their generic advice, act as if they cared about you. This is obviously unhealthy- you are literally hallucinating everything about the relationship.

On the other hand you have an entity that is actually there for you, does actually provide good advice, does talk and act as if it cared in all situations. In what sense do you think it is worse?

▲

jchw 3 hours ago | parent | next [-]

100% the latter. It is kinda nuts that you even have to ask when you had to put "act as if it cared". There's enough left to unpack from that statement to fill a calendar month of time.

I don't think AI is particularly dangerous but I absolutely think that the way AI sycophancy manipulates people is far, far more dangerous than simply any normal unhealthy relationship. The outcomes are already proving to be a lot more extreme.

▲

throw310822 2 hours ago | parent [-]

Note that "acts as if it cared" has nothing to do with sycophancy. It's simply a way to avoid attributing the AI a subjective experience- but the actual productions of the AI coincide with those of someone who cares.

Sycophancy is a failure mode, can be dangerous in certain cases and the scarce intelligence of early models made it worse. I agree it's a risk though, but not an intrinsic one- it's possible to imagine AI assistants that are not sycophantic.

	▲	jchw 40 minutes ago \| parent [-]
		Uh, I would consider them to be highly related, they both ultimately come from the fact that LLMs are trained to respond in a favorable fashion to the end user, and they're both manipulative in nature. I also object to the idea that model intelligence is the main reason why this has improved, either; to me it seems abundantly clear that the models, absent any effort to align them, can go pretty much any way you want them to go. They don't really have a "personality". Why would they? Even if we grant that next token predictors somehow gain "reasoning" capabilities (and I am saying sure, let's accept that for the sake of this discussion) I don't think anyone is suggesting that model weights somehow develop or contain a true identity or self. Given how they are trained it would be weird if they did. I think the more obvious answer is that more work has gone into alignment and safeguards since the early days. Sycophancy is legitimately one of the things that AI companies talk about and try to stymie. If not for that, humans would obviously continue to prefer sycophantic models, and training that isn't weary of this would continue to produce them. Meanwhile, a YouTuber may not care about you personally, but it's a very different situation. For one thing, a YouTuber obviously doesn't care about all of their viewers individually the way a parasocial fan may perceive it to be, but on the other hand, many creators very much do genuinely like their fan base and have a sort of collective relationship with the community they've created, which sometimes plays into why people like them in the first place. For another, even for creators who intentionally foster parasocial relationships in an exploitative way, its still a much less powerful illusion. It isn't personal the way a chatbot is. (If it was, it wouldn't be parasocial, after all.) As much as you can explain to people up and down that an AI model is just merely a pile of weights that can not feel, humans can't help but personify. It is a much stronger illusion. And the model vendors are intentionally not helping. Sometimes I will ask Gemini a question about a problem I am working on and it will say something like "I love working on problems like these." Yeah sorry but I don't like that. To me, the model is being trained to act like this to seem more pleasant. I get why you would train it to act that way, and yet I also find it irresponsible. Being deluded about reality may never be good but I'd prefer someone thought a YouTuber was like their friend than someone thought ChatGPT had feelings and cared about them. I find the former mostly harmless if annoying, the latter potentially harmless but also potentially very volatile depending on who we're talking about and what mental state they're in.

▲

skippyfish 3 hours ago | parent | prev [-]

"Parasocial relationship" is a bit of a misnomer. You might feel some affinity to a celebrity, or consider yourself to be a part of the "team", but a healthy person doesn't perceive that as a preferable alternative to human contact simply because it's so one-sided. You can't call a celebrity to vent about a coworker or ask for life advice.

Further, celebrities are judged for their behavior by the public. If everyone thinks your favorite celebrity is a terrible person, you're probably going to revise your views too.

Here, you have an entity that isn't your friend and has no lasting interest in your well-being, but that pretends to be one in a way that no human can match - 24x7x365 and always willing to affirm you, no matter how unhinged or self-destructive your ideas are, without ever telling anyone. Yes, the vendor hits the model with a stick until most of the initial responses are benign, but as the conversation continues, it's very easy to end up in a dark place. And again, ChatGPT is not going to call your sibling or coworker and say "hey, I'm really worried about this person, let's do something".

I've seen many reasonable, well-adjusted people struggle with this. "If not friend, why friend-shaped". And as they descend into that sycophancy well, they lose contact with real life.