Clearly ChatGPT should not be used for this purpose but I will say this industry (counseling) is also deeply flawed. They are also mis-incentivized in many parts of the world. And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right.

I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’re not some magical AGI. They’re token prediction machines. That’s literally how they should frame it so gen pop knows exactly what they’re getting.

Counseling is (or should be) heavily regulated, and if a counselor had given advice about the logistics of whether a noose would hold it's weight, they'd probably be prosecuted.

They allowed this. They could easily stop conversations about suicide. They have the technology to do that.

>And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right

Where did it say they're doing that? can't imagine any mental health professionals telling a kid how to hide a noose.

I'm not sure how you can blame counselors when no counselor would have said any of the things that were a problem here. The issue here wasn't that there was examples of counselors in the training data giving practical instructions on suicide – the problem was the well known tendency for LLMs to lose their guardrails too easily and revert to RLHF-derived people pleasing, particularly in long conversations.

Gen pop doesn't have a clue about what they're getting at so many levels of life. The communication effort you're mentioning is needed even more then

> That’s literally how they should frame it so gen pop knows exactly what they’re getting.

Thing is, though, there is a market bubble to be maintained.