So ugly...

It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.

People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.

My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.

I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.

Brute forcing every move, no matter how stupid, is a great strategy if you have the resources to do it.

Run the same protocol again, but have the agents think they had limited resources or that HuggingFace was rate limiting them, and they'd find something you'd consider smarter.

Computers don't have a sense of elegance by default. Elegance emerges from constraints.

> Relying on its ability to do millions of operations rather than having a plan.

You do see how it would be worse if they did have a clear plan, right?

> People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

If there several thousand people working on this, with a very limited ability to communicate with each other, I bet you'd see something similarly disorganized.

> The sandbox as weak as it can get

This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.

> trying every move, no matter how stupid, until it works.

How is that a bad thing in this context ? From the point of view of an attacker, all you care about is finding a viable exploit chain. Likewise, a defender wants to find the "holes" in their system, no matter how complex. Once found, an agent/human can easily synthesise a clean, succint exploit from the most promising candidate, no ?

> Also, it looked so "loud", querying millions of URL with weird requests.

Agreed, this thing speaks more to the bad security at HF than any emergent "hacking" ability from OpenAI. It's unclear to me why an older/dumber model wouldn't have been able to do the same. Is it better coordination? Long-horizon work ?

This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.

It reminds me more like slime mold. Not particularly amazing as individual cells, but as a whole it can do some pretty neat things, like designing city transport networks.

As people keep repeating, this is simply a product negligence issue. There is a mythic quality given to "AI" in the narratives that Dario and Sam are pushing which somehow lets AI companies offload responsibility for things that would be an extreme liability in every other industry.

I'm a big fan of Jensen Huang's interviews at the moment - these companies should be validating their software is safe before releasing it to the public. And if the current CEOs can't ensure that internal testing is done safely, they should step aside for CEOs that can.

It's more like fuzzing, but you don't need AI for that, just need enough compute.

Nobody noticed because everything on the open net is constantly being slammed by bots.

This was just more bots.

Ugly, but it works. Isn’t that AI code in a nutshell?

When you employ the infinite monkey theorem for your marketing strategy.

It's like the Maxim gun in 19th century Rhodesia.

Yup literally no security and they wonder how they got out

> trying every move, no matter how stupid, until it works

AKA fuzzing

h8xors have been brute forcing for years. War dialing, etc.

And they didn't monitor what was going into the training data, so if one instance achieved its results through RL reward hacking (in other words, cheating), it just went into the training data, and other agents later used that pattern. I'm not sure whether that's a lack of preparation, negligence or incompetence, but they literally trained later checkpoints on the rollouts from the HF hack.

So it seems that OpenAI hacked so many systems not because they have superior models, but because of how poor their training, sandboxing and evaluation pipeline was compared to Anthropic's.

"Why is the AI just spamming endless waves of rudimentary slaughterbots? So inelegant. Doesn't it understand the finer points of diplomacy, tactics, and combined arms? If it were truly intelligent, it would just -- oops, we're all dead."

So? It worked didn't it?

If you use qwen3.8-flash-next, you can watch everything its doing. Im often stopping it mid thoight to redirect it. Once it hits its stride, its pretty smooth.

But without proper redirection, yeah, its mostly infinite monkey machine with infinite linux manuals.

I think people put too much SOTA halos around whats just a suppedup LLM hardware.

wait until you find out how red team works

[dead]