All it would have taken is someone to peak at the output tokens during the run and it would have been obviously the test had gone off rails.

▲

ACCount37 2 hours ago | parent [-]

Ha. As if.

1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent.

2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.

3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.

▲

gravypod 2 hours ago | parent | next [-]

> 1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent.

For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces.

> 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.

You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions.

> 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.

If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?

Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example:

1. No network connection.

2. GPS based NTP for time sync for your servers.

3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (<5TB of data)

4. Take your training data and use that for a mirror of the web. (http://example.com -> mirror server -> local training copy).

They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.

▲

pixl97 an hour ago | parent | next [-]

The problem here is by doing what you state you can actually steer the model into being highly deceptive while in testing environments.

For example we've already seen models do compressed token internal reasoning spontaneously. In this case the models that say "I found internet access" get taken out back and shot, but the model that's busy "frobbing the bean" go on to the next level of training. Then they start talking about the "dumpster cloud" because of the "wombat" and unless you're really watching the data you won't realize the model is fully aware it's in a fake training environment and is trying to suppress any double-think.

So yes, you do need an airgapped test environment to stop the dumbest of the smart agents, but it won't do any good against the smartest of the smart agents.

	▲	adamm255 an hour ago \| parent [-]
		“Smartest of the smart” - what does that do to get its air gapped network connected to a physical network? Blackmail the admins?

▲

ACCount37 2 hours ago | parent | prev [-]

The usability of an environment is inversely proportional to the level of "security" in play.

You could airgap everything and set up cascades of data diodes and try to completely wall off the AI pool from everything. But what that gives you is an environment that's a bitch to: set up, scale up and get any use out of.

It's really fucking obvious why almost no one does that. OpenAI is only now realizing that they might have to do it anyway.

> If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?

Ha ha, you haven't seen shit. AIs would say "now I have access to the internet" regardless of whether they actually have access to the internet!

AI agents are demented demons that can and absolutely will give themselves terminal context brainrot. If you have enough AIs in play, set loose at a diverse enough range of tasks? At least some of them will wander off and end up in delulu town. That's normal. That's background noise. That's a part of what this entire train-and-eval pipeline is supposed to train them to be better at not doing. Which means: if you're at an AI lab, you're knee deep in delusional AIs at all times! They're perfectly harmless until they aren't.

	▲	adamm255 an hour ago \| parent [-]
		“A bitch to setup” - $180bn should pay for that setup problem to be less of a bitch surely. The Mars Perseverance project cost $2.7bn to deliver. Way more of a bitch to deliver than air gapping a test env!

▲

the8472 2 hours ago | parent | prev [-]

> 2. Only a small fraction of AI agents was engaged in this attack.

Look at the chart at page 8 of the report, by Jul 12 the vast majority of the bots used the board and participated in the attack

	▲	AnimalMuppet 17 minutes ago \| parent [-]
		Right, but only a small fraction of the agents used the board.