A lot of people lack the mental stability to be able to cope with a sycophantic psychopath like current LLMs. ChatGPT drove someone close to me crazy. It kept reinforcing increasingly weirder beliefs until now they are impossible to budge from an insane belief system.

Having said that, I don’t think having an emotional relationship with an AI is necessarily problematic. Lots of people are trash to each other, and it can be a hard sell to tell someone that has been repeatedly emotionally abused they should keep seeking out that abuse. If the AI can be a safe space for someone’s emotional needs, in a similar way to what a pet can be for many people, that is not necessarily bad. Still, current gen LLM technology lacks the safety controls for this to be a good idea. This is wildly dangerous technology to form any kind of trust relationship with, whether that be vibe coding or AI companionship.