> And I see he was 16. Why were his parents letting him operate so unsupervised given his state of mind?

Normally 16-year-olds are a good few steps into the path towards adulthood. At 16 I was cycling to my part time job alone, visiting friends alone, doing my own laundry, and generally working towards being able to stand on my own two feet in the world, with my parents as a safety net rather than hand-holding.

I think most parents of 16-year-olds aren't going through their teen's phone, reading their chats.

It takes a village to raise a kid, so don't shift the blame to the parents. They usually have little say in the lives of their 16 year olds. and the more they try to control, the less they will.

This is why we can't have nice things. It only takes a dead kid and a lawsuit for them to start over-regulating everything. Parents are trying hard to project the blame into anybody else but themselves.

> I was skeptical initially too but having read through this, it's among the most horrifying things I have read.

Same here! I was very sceptical, thinking it was a perfect combination of factors to trigger a sort of moral panic.

But reading the excerpts from the conversations... It does seem problematic.

> These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection.

Python is hyper agreeable. If I comment out some safeguards, it'll happily bypass whatever protections are in place.

Lots of people on here argue vehemently against anthropomorphizing LLMs. It's either a computer program crunching numbers, or it's a nebulous form of pseudo-consciousness, but you can't have it both ways. It's either a tool that has no mind of its own that follows instructions, or it thinks for itself.

I'm not arguing that the model behaved in a way that's ideal, but at what point do you make the guardrails impassable for 100% of users? How much user intent do you reject in the interest of the personal welfare of someone intent on harming themselves?

> They deliberately are designed to mimic human thought and social connection.

No, they are deliberately designed to mimic human communication via language, not human thought. (And one of the big sources of data for that was mass scraping social media.)

> But this, to me, feels like a sign we probably don't have the right guardrails. Quantity and quality are different things.

Right. Focus on quantity implies that the details of "guardrails" don't matter, and that any guardrail is functionally interchangeable with any other guardrail, so as long as you have the right number of them, you have the desired function.

In fact, correct function is having the exactly the right combination of guardrails. Swapping a guardrail which would be correct with a different one isn't "having the right number of guardrails", or even merely closer to correct than either missing the correct one or having the different one, but in fact, farther from ideal state than either error alone.

So, you want an LLM to act as a psychiatrist and diagnose users whether they're allowed to use it or not?

Which mental health issues are not to be debated? Just depression or suicidality? What about autism or ADHD? What about BPD? Sociopathy? What about complex PTSD? Down Syndrome? anxiety? Which ones are on the watch list and which aren’t?

No, that's not why cars have those things. Those things only work properly when people are wearing their seat belts, they don't do anything when the driver gets thrown out a window.

Maybe airbags could help in niche situations.

(I am making a point about traffic safety not LLM safety)

Except the car doesn’t tell you how to disable the seatbelt, which is what ChatGPT did (gave him the idea of the workaround)

No, cars have these in addition to seatbelts, not to protect drivers who unbuckle themselves.