My understanding of LLMs with attention heads is that they function as a bit of a mirror. The context will shift from the initial conditions to the topic of conversation, and the topic is fed by the human in the loop.

So someone who likes to talk about themselves will get a conversation all about them. Someone talking about an ex is gonna get a whole pile of discussion about their ex.

... and someone depressed or suicidal, who keeps telling the system their own self-opinion, is going to end up with a conversation that reflects that self-opinion back on them as if it's coming from another mind in a conversation. Which is the opposite of what you want to provide for therapy for those conditions.

▲

citizenpaul 24 minutes ago | parent | next [-]

The real question to me here is not the computer. Its why is there such a segment of the population that is so willing to listen to a machine? It it upbringing, societal, circumstance, mental health, genetic?

I know the Milgram obedience to authority experiments but a computer is not really an authority figure.

▲

layman51 2 hours ago | parent | prev [-]

In a way this kind of reminds me of how in some religions or cultures, they may try to warn you away from using Oujia boards or Tarot, or really anything where you are doing divination. I suppose because in a way, it could lead to an uncharted exploration of heavy topics.

I’m not a heavy user of LLMs and I’m not sure how delusional I could be, but I wonder if a lot of these things could be prevented if people could only send like one or two follow up messages per conversation, and if the LLM’s memory was turned off. But then I suppose this would be really bad for the AI companies’ metrics. Not sure how it would impact healthy users’ productivity either. Any thoughts?

	▲	shadowgovt 2 hours ago \| parent [-]
		Not just the metrics, the actual utility. For the things the LLMs are good at, the context matters a lot; it's one of the things that makes them more than glorified ELIZA chatbots or simple Markov chains. To give a concrete example: LLMs underpin the code editing tools in things like Copilot. And all that context is key to allow the tool to "reason" through the structure of a codebase. But they should probably come with a big warning label that says something to the effect of "IF YOU TALK ABOUT YOURSELF, THE NATURE OF THE MACHINE IS THAT IT WILL COME TO AGREE WITH WHAT YOU SAY."