I was under the impression that all the mainstream models already do this. I'm sure you could probably download some obscure, uncensored and unhinged model that says anything you want, but that isn't what 99% of people will be interacting with.

Not strictly relevant but I also have concerns about AI psychosis which seems related a little bit here, otherwise they'd realise it's a computer program and can't make you do anything.

I think it would be better if children learned critical thinking. It would help defend against any unsound conclusions proposed AI or any other sources.

https://en.wikipedia.org/wiki/Critical_thinking

Obviously, they already tried.

Problem is that there simply is not a way to do this reliably. The models are all stochastic processes and the only real levers model designers have to pull involve asking the model to pretty please not do something bad.

And then it turns out that it's pretty easy to also ask models to pretty please ignore previous instructions. You can also accidentally get a model into a state where it ignores system prompt guidelines.

There is not a big #ifdef DONT_TELL_USER_TO_DIE switch in the code. Nobody truly understands how the models work under the hood and there simply is not a way to enforce 100% that a model cannot do something.