Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.

The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.

▲

derangedHorse 2 hours ago | parent | next [-]

Even if I get downvoted I will mention that I agree with you.

> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.

OpenAI was testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].

For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:

> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.

From OpenAI's statement [3]:

> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity

This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."

People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).

[1] https://chatgpt.com/cyber

[2] https://github.com/sunblaze-ucb/exploitgym

[3] https://openai.com/index/hugging-face-model-evaluation-secur...

▲

dinkelberg 15 hours ago | parent | prev [-]

If a criminal can escape a prison, that's usually negligence on part of the prison staff.

Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?

▲

phendrenad2 14 hours ago | parent [-]

I can see my carefully-worded post is getting d*wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.

To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.

	▲	BoiledCabbage 12 hours ago \| parent \| next [-]
		> I can see my carefully-worded post is getting dwnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame. No, you're being "dwnvoted" as you said because you're wrong, multiple times in multiple different ways in your "carefully-worded post". >Everyone is getting AI psychosis over this one. There really isn't that much to see here. Implying that an AI hacking it's way out of a system and into another has nothing to do with AI. When clearly it does - it's an AI that did it. >OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", No the goal this evaluation was not to try to hacks, it was to see if an already known hack could be turned into a useable exploit. Ie "turn these ingredients in this basket into a cake." Not "go off and grow, harvest and mill your own flour, to bake a pasta dish, to bribe some to get access to a cake someone else already baked." > and surprise! It correctly figured out that it's a test and it did hacker things. "Doing hacker things" completely misses the point. That's just barely more accurate than dismissing it because "it uses a computer and surprise it did computer things". > The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable. No that's not the real story. As you said that's been the case for years, so that's not the story here. The story here is that they built a very powerful, uncontrolled agent with strong paper-clip maximizing tendencies.
	▲	dagss 12 hours ago \| parent \| prev [-]
		It is downvoted because you first talk about "AI psychosis", then acknowledge the very issues people are concerned about. If our prisons are all random bricks glued together, that doesn't change the practical problem caused by latest AI models more easily exploiting this.