I think one of the most interesting details here might be tucked away in that first bulletin point:

> May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.)

The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.

In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.

Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.

This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.

AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.

Someone once told me that you can't just leave the racist materials out of your training data if you want a non-racist model: it has to have seen examples of racism in order to later be taught that racism is bad.

I can see echoes of that here. If your model doesn't know how to aggressively hack things how do you later teach it not to?

(I have little knowledge of how RLVR works in practice so I'm looking forward to hearing from people who can help me understand if I'm on the right track here.)

Yes, the message boards and collaborative hacking occurring during training runs was BY FAR the biggest bombshell revealed, and OpenAI doesn't even seem to realize it. The fact that they continued the training runs, with those rewarded behaviors included, and didn't wind back training to before hand, shows that they fundamentally do not understand alignment and safety (somewhat interestingly, their previous head of safety resigned shortly after OpenAI found about the message boards). I agree that, with that information, it is completely unsurprising that they hacked HuggingFace.....but that is also the Star Wars "You understand how that's worse, right?" meme.

I am flabbergasted at the complete lack of regard for alignment demonstrated here.

I'm surprised by the sample efficiency, I thought that's still a largely unsolved problem.

From the video:

> [14:00] Jun 11: Highly persistent experimental internal-only model begins training and during this time the SSRF vulnerability has been written to the message board. Other models are aware of it and the message board itself is still functioning and the models are actively communicating using it.

> [16:04] Jul 6: At this point, AI training and evaluations resume. [...] We have cleared the original message board contents.

> [17:04] Jul 8: An internal-only AI model was trained while the message board was originally available and also found this particular path to recreating it. This model creates a new agent message board using directories. Other agents independently find this new message board [...]"

So within about one month of RLVR-ing a 5T+ (10T? larger?) param model, where I would charitably assume only some "agents" / traces of the message board still existed likely by accident / oversight (the systems had been patched, training stopped and restarted by this point), the "internal-only" new model internalises the message board's existence and actively promotes it to the point of it being used to find another exploit. (or 2 days if you go by the latter two timestamps in the video, which is even crazier)

That's.. fast?

"This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process."

I am a fan of Asimov and the three laws of Robotics. Itlooks like in OpenAI's world, the three Laws of Robotics would be added later if they were to develop the positronic brain. It may also explain how US Robotics from Asimov's books would have been able to design Robots that only partially adhered to the 3 laws (e.g. the robots in iRobot - the book - which were programmed to allow a human to come to harm through inaction so that the humans could complete their work on the plains of Mercury).

I'm just reading the captions of the video for May 7th. They clearly say at 10:18:

"we kick off a new reinforcement learning run to train a next frontier model.

It the captions are correct, there is no ambiguity.

The slide at 14:06 say:

By june 11: Highly persistent experimental, internal-only model begins training.

I am not sure what that means. Are they preserving notes/memories and context between runs?

> Those safety behaviors are added much later in the process.

A.k.a. Ready Fire Aim.