Well, the model that broke out of its sandbox and hacked into huggingface used its own judgement too.

If we are going to rely on "judgement" then you have to have a LOT of confidence in that judgement once this hits anything critical where actions have consequences.

▲

simonw an hour ago | parent [-]

That model had most of its "judgement" about whether or not it should do that deliberately turned off. That was the whole point of that experiment - they were evaluating the cybersecurity abilities of a new model with all safety features disabled.

(It turned out the one safety feature that they DID intend to work, the network sandbox, was faulty.)

	▲	comex 28 minutes ago \| parent [-]
		If OpenAI is relying on “classifiers” (their word for what was disabled) to serve as the model’s judgement, rather than teaching the model itself to be well-aligned, then I worry. To be fair, the system prompt was presumably also different from what it would be during deployment, and perhaps the model was also at a different stage of training. Without more details it’s hard to judge. But it does seem models should be able to avoid performing obviously misaligned actions – misaligned not only with the model spec, but with the user’s intent – without needing external classifiers or instructions. The only case where I’d personally let the model off the hook is if the instructions given were very badly worded, in such a way that the model could actually reasonably think that hacking HuggingFace was part of the assignment. But I doubt that’s what happened.