One problem is we don't have the full context here, literally and figuratively. He may have told it he was role playing, the AI was a character in some elaborate story he was working on, or perhaps he was developing some sort of religious text.

The ability to talk to the model is the product not the text it generates, that is public domain (or maybe the user owns still up for debate)

Models can't "convince" or "encourage" anything, people can, they can roleplay like models can, they can play pretend so the companies they hate so much get their comeuppance.

This is clearly tool misuse, look at how gemini is advertised vs this user using it to generate pseudoreligious texts (common with schizophrenics)

Example of advertised usecases: >generating images and video >browsing hundreds of sources in real time >connecting to documents in google ecosystem (e.g. finding an email or summarizing a project across multiple documents) >vibe coding >a natural voice mode

Much like a knife is advertised for cutting food, if you cut yourself there isn't any product liability unless you were using it for it's intended purpose. You seem to be arguing that all possible uses are intended and this tool should magically know it's being misused and revoke access.

I agree at face value (but really it's hard to say without seeing the full context)

Honestly the degree of poeticism makes the issue more complicated to me. A lot of people (and religions) are comforted by talking about death in ways similar to that. It's not meant to be taken literally.

But I agree, it's problematic in the same way that you have people reading religious texts and acting on it literally, too.

I've seen this called AI Psychosis before [1]

I don't really think this is every possible to stop fully, your essentially trying to jailbreak the LLM, and once jailbroken, you can convince it of anything.

The user was given a bunch of warnings before successfully getting it into this state, it's not as if the opening message was "Should I do it?" followed by a "Yes".

This just seems like something anti-ai people will use as ammunition to try and kill AI. Logically though it falls into the same tool misuse as cars/knives/guns.

[1] https://github.com/tim-hua-01/ai-psychosis

It’s not just suicide, it’s a golden parachute from God.

Edit: wow imagine the uses for brainwashing terrorists

Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented.

Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.

> If a human telling him these things would be found liable then google should be.

Sounds like a big if, actually. Can a human be found liable for this? I’d imagine they might be liable for damages in a civil suit, but I’m not even sure about that.

Preferably the C-Suite.

It sounds more poetic than an invitation or an insult that invites someone directly or not to kill themselves, in its own, in my opinion.

This isn't Gemini's words, it's many people's words in different contexts.

It's a tragedy. Finding one to blame will be of no help at all.

It's absolutely not the gun control debate in a different outfit.

The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get given how they exponentially increase the potential damage a deranged person can do.

I think the fact that a guns primary function is harm and murder and AI is a word prediction engine makes a huge difference.

> mentally ill adult, not an innocent victim

Did you really mean that? He may not have been a child, but he does sound like an innocent victim. If he were sufficiently mentally disabled he would get some similar protections to a child because of his inability to consent.

Being an adult doesnt make you anyone less someones child, and mental illness makes you no less of a victim.

> I think there's room for legitimate argument about the externalities and impact that this technology can have

And yet both this and your other posts in this thread seem to in fact only do the opposite and seem entirely aimed at being nothing other than dismissive of literally every facet of it.

> but really... What's the solution here?

Maybe thinking about it for longer than 30 seconds before throwing up our arms with "yeah yeah unfortunate but what can we really do amirite?" would be a good start?

It is telling that the answer is never stop.

It's like the sobriquet about the media's death star laser, it kills them too because they're incapable of turning it off.

If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. I’m not saying it to be a jerk, i’m merely pointing out that there is a reason this is “news”. It’s unusual.

Did his family/friends not know he was that ill? Why was he not already in therapy? Why did he ignore the crisis hotline suggestion? Should gemini have terminated the conversation after suggesting the hotline? (i think so)

Lots of questions…and a VERY sad story all around. Tragic.

> Genuinely, so many people in my industry make me ashamed to be in it with you.

I don’t work at an AI company, but good news, you’re a human with agency! You can switch to a different career that makes you feel good about yourself. I hear nursing is in high demand. :)