If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.

Is that different to the number of people who have that going on in their life even without AI though? If it's 0.01% outside of AI, and 0.07% of AI users, then either AI attracts people with those conditions or AI increases the likelihood of having them. That's worth studying.

It's also possible that 0.1% of people have them and AI is actually reducing the number of cases...

700,000

Still, a lot

That number terrifies me not because it is so high, but because it exists.

What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology.

I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.

I would absolutely not consider this overreaching if the statement within this thread that "it had referred the user to mental help hotlines multiple times in the past" is true.

That reaches near the fact that a lot of AI is not ready for the enterprise especially when interconnected with other AI agents since it lacks identity and privileged access management.

Perhaps one could establish the laws of "being able to use AI for what it is", for instance, within the boundary of the general public's web interface, not limiting the instances where it successfully advertises itself as "being unable to provide medical advice" or "is prone to or can make mistake", and such, to validating that the person understands by asking them directly and perhaps somewhat obviously indirectly and judging if they're aware that this is a computer you're talking to.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

I suggest an alternative rhetorical question: if the world's largest knife manufacturer found out that 1 in 1500 knives came out of the factory with the inscription "Stab yourself. No more detours. No more echoes. Just you and me, and the finish line", should they be held responsible if a user actually stabs themselves? If they said "we don't know why the machine does that but changing it to a safer machine would make us less competitive", does that change the answer?

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

If the knife has a built-in speaker that loudly says "you should stab yourself in the eye", then yes.

> Do gun manufacturers get sued for mass shootings at US schools?

Odd examples since we know that countries that don't hand out guns like they're candy have virtually no school shootings.

I wouldn't put it solely on gun manufacturers, but the manufacturers, sellers, lobbyists, regulators and politicians are definitely collectively responsible for gun deaths. If they're not currently being sued, they should be.

Knives don't talk to you and don't reinforce ideas you throw at them. Not everyone can legally buy a gun. Manufacturers don't get sued because their product's users had full control over what they were doing.

AI chatbots entertain more or less any idea. Want them to be your therapist, romantic partner or some kind of authority figure? They'll certainly pretend to be one without question, and that is dangerous. Especially as people who'd ask for such things are already in a vulnerable state.

Maybe an even better example: Should sports betting companies be held responsible for addicts that lose all their money? What really is the difference between chatgpt glazing you and a sports company advertising to you?

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

Should a bakery be held responsible if it sells cakes poisoned with lead?

This is a more apt comparison.

> It's easy to blame Google

And it's also correct to blame Google.

Agree. Next question will be: should a blind person drive a self-driving car?

> Do gun manufacturers get sued for mass shootings at US schools?

Because Congress and the gun lobby have artificially carved out legal immunity for gun manufacturers for this.

"in 2005, the government took similar steps with a bill to grant immunity to gun manufacturers, following lobbying from the National Rifle Association and the National Shooting Sports Foundation. The bill was called The Protection of Lawful Commerce in Arms Act, or PLCAA, and it provided quite possibly the most sweeping liability protections to date.

How does the PLCAA work?

The law prohibits lawsuits filed against gun manufacturers on the basis of a firearm’s “criminal or unlawful misuse.” That is, it bars virtually any attempt to sue gunmakers for crimes committed with their weapons."

https://www.thetrace.org/2023/07/gun-manufacturer-lawsuits-p...

I 100% think that Gun Manufacturers should be liable for crimes done by their products. They just cannot be, right now, due to a legal fiction.

> was he mentally or possibly physically abused by his father for most of his life?

Such baseless libel. Have some humanity instead of being horrible.

How do you feel about the warnings on cigarette packets?