The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.

All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).

Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.

My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).

I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.

Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.

▲

manux 36 minutes ago | parent | next [-]

Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).

▲

simonw 33 minutes ago | parent [-]

I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.

OpenAI said:

> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.

It was a test of raw capabilities of the underlying model.

▲

manux 20 minutes ago | parent | next [-]

I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.

I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.

▲

simonw 12 minutes ago | parent [-]

I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.

A model that can do that is aligned with me.

The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)

Even AGI doesn't give you a model that can read minds and forecast the future.

	▲	manux 6 minutes ago \| parent \| next [-]
		But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not) That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
	▲	dist-epoch a minute ago \| parent \| prev [-]
		The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam. It knew exactly what it was doing. It was not asked to cheat the CyberGym by stealing the answers. This is text book misalignment. If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail to search your email, just in case you were discussing some possible vulnerabilities of your software with someone?

▲

mjamesaustin 24 minutes ago | parent | prev | next [-]

Alignment isn't alignment if it can be turned on and off at the whim of company employees.

This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.

	▲	verdverm 9 minutes ago \| parent [-]
		Alignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.

▲

joe_the_user 15 minutes ago | parent | prev [-]

This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).

The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).

Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.

▲

efromvt 2 hours ago | parent | prev | next [-]

Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.

▲

tptacek 2 hours ago | parent [-]

I think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.

▲

notaharvardmba an hour ago | parent [-]

Also, that’s the wrong mental model. Anyone who runs a sass platform or a website knows that the Internet is already full of millions and millions and millions of bots and scripts and other random shit that’s always trying to attack you, often completely randomly. Security is always a battle between good and evil. All I can say is that if you’re in charge of keeping something secure, you should probably try to get your hands on the best tools to do that. I think the problem in this case is that the best tools to do that are also the tools that are making it more easy for evil to occur. So it’s more for me a perception of someone creating a threat and the tool to fix it and then charging for it which doesn’t feel right.

	▲	pishpash 27 minutes ago \| parent [-]
		That's exactly what they are doing. Out of all the civilian applications that AI capabilities have, why has this surfaced as a priority for demonstration? Probably because of money.

▲

indigodaddy 39 minutes ago | parent | prev | next [-]

re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)

▲

futureshock 2 hours ago | parent | prev [-]

I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.