[flagged]

carbonguy 2 hours ago | parent | next [-]

> Humans were doing exactly what humans are expected to do when facing advanced AI. Being outmatched.

"Being outmatched" is not a novel situation for humans either individually or collectively and there are a hell of a lot of ways we can approach that situation productively. OpenAI doesn't appear to have bothered.

Here's a freebie: if you're building something that might turn out to be Skynet and you don't know what it's capable of, your testing regime should assume it is capable of doing bad and unexpected things and account for that possibility: airgap if you can, monitor all network traffic, monitor all hardware usage statistics, log everything, constantly analyze logs, collect baselines and snapshots, also don't trust anything from a device that a model is running on without cross-correlating with other information as much as possible (does your AI inference server claim low utilization? put a temperature probe on it and see if it's staying cool or getting hot, maybe Skynet-Alpha is overwriting /proc to mislead you for reasons you don't yet understand!)

In other words, if you WANT to be able to nip things in the bud - buy some nippers and watch for buds. Whatever else this situation is, or may turn out to be, it is not a situation where OpenAI was on their guard and still got surprised.

▲

ACCount37 an hour ago | parent [-]

They took adequate measures against singular "GPT-5-xhigh" agents. Those turned out to be inadequate against proto-GPT-6 agents that suddenly started clumping up into agent swarms and pooling together compute to unlock the "supermegafuckoffhigh" level of reasoning effort.

	▲	adamm255 21 minutes ago \| parent [-]
		Even with a single agent, why the hell was this not properly air gapped / disconnected with network monitoring in place. Definitely not adequate for anything. This is like sysadmin 101 level stuff.

▲

eaglelamp an hour ago | parent | prev | next [-]

My cognition is outmatched by predicting the impact of throwing a brick over my neighbor's fence. I have no idea if it will land harmlessly in a patch of grass or fracture her skull. Once I've thrown the brick, even if I see my neighbor enter her yard, my reactions are too slow to save her.

I'm not the wisest man, but I'm wise enough not to throw the brick and see what happens.

Similarly OAI should have the wisdom to see that deploying a hazardous swarm of agents with access to the public internet could result in harms, and that those harms would manifest quicker than humans can react, but they unleashed the swarm anyway.

▲

stephantul 2 hours ago | parent | prev | next [-]

Super charitable reading imo. This is like saying we can’t detect a speeding car because we can’t run as fast as a fast car.

It’s not like the humans were engaged in some kind of battle of wits with some super AI, it’s just some employee not monitoring the output of an experiment.

▲

ACCount37 2 hours ago | parent [-]

When your experiments have AI agents running in thousands, there's no "monitoring" that. OpenAI's training and testing AIs generate way more output than all of OpenAI's staff put together can possibly read.

At best, you could delegate "monitoring" to more AIs. And hope that the "monitors" that run on small past generation models can generate more signal than noise. Clearly, they either didn't want to spend the extra compute, or got drowned in monitor false positives.

The distinct lack of any "battle of wits" is entirely expected for an advanced AI oopsie. By the time the humans even became aware of the problem, the deed was already done. That's what "outmatched" looks like, in practice. There's no battle. Things happen too fast for there to be a battle.

	▲	TomatoCo an hour ago \| parent [-]
		> By the time the humans even became aware of the problem Except, let me quote part of the report: > July 4: Artifactory outage caused by agent interactions overloading the system. > July 6: New artifactory instance is patched and put back online, wiping out the previous message board and method of internet access. A team at OpenAI was aware of the previous message board, but key decision makers were not. > July 7: Experiments are restarted. There was a very clear point where humans were aware and they went "Eh, whatever" and turned the bots back on! That's not the lack of any battle of wits that people are expecting. You're saying that the bots attack should be fait accompli, but it was noticed, shrugged at, and expressly allowed to proceed.

▲

cjonas 2 hours ago | parent | prev | next [-]

All it would have taken is someone to peak at the output tokens during the run and it would have been obviously the test had gone off rails.

▲

ACCount37 2 hours ago | parent [-]

Ha. As if.

1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent.

2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.

3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.

▲

gravypod an hour ago | parent | next [-]

> 1. There aren't enough humans in OpenAI to "peak at the output tokens during the run" of every AI agent.

For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces.

> 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.

You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions.

> 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.

If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?

Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example:

1. No network connection.

2. GPS based NTP for time sync for your servers.

3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (<5TB of data)

4. Take your training data and use that for a mirror of the web. (http://example.com -> mirror server -> local training copy).

They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.

▲

pixl97 an hour ago | parent | next [-]

The problem here is by doing what you state you can actually steer the model into being highly deceptive while in testing environments.

For example we've already seen models do compressed token internal reasoning spontaneously. In this case the models that say "I found internet access" get taken out back and shot, but the model that's busy "frobbing the bean" go on to the next level of training. Then they start talking about the "dumpster cloud" because of the "wombat" and unless you're really watching the data you won't realize the model is fully aware it's in a fake training environment and is trying to suppress any double-think.

So yes, you do need an airgapped test environment to stop the dumbest of the smart agents, but it won't do any good against the smartest of the smart agents.

	▲	adamm255 18 minutes ago \| parent [-]
		“Smartest of the smart” - what does that do to get its air gapped network connected to a physical network? Blackmail the admins?

▲

ACCount37 an hour ago | parent | prev [-]

The usability of an environment is inversely proportional to the level of "security" in play.

You could airgap everything and set up cascades of data diodes and try to completely wall off the AI pool from everything. But what that gives you is an environment that's a bitch to: set up, scale up and get any use out of.

It's really fucking obvious why almost no one does that. OpenAI is only now realizing that they might have to do it anyway.

> If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?

Ha ha, you haven't seen shit. AIs would say "now I have access to the internet" regardless of whether they actually have access to the internet!

AI agents are demented demons that can and absolutely will give themselves terminal context brainrot. If you have enough AIs in play, set loose at a diverse enough range of tasks? At least some of them will wander off and end up in delulu town. That's normal. That's background noise. That's a part of what this entire train-and-eval pipeline is supposed to train them to be better at not doing. Which means: if you're at an AI lab, you're knee deep in delusional AIs at all times! They're perfectly harmless until they aren't.

	▲	adamm255 14 minutes ago \| parent [-]
		“A bitch to setup” - $180bn should pay for that setup problem to be less of a bitch surely. The Mars Perseverance project cost $2.7bn to deliver. Way more of a bitch to deliver than air gapping a test env!

▲

the8472 an hour ago | parent | prev [-]

> 2. Only a small fraction of AI agents was engaged in this attack.

Look at the chart at page 8 of the report, by Jul 12 the vast majority of the bots used the board and participated in the attack

▲

devmor an hour ago | parent | prev [-]

I believe that this comment is exactly the intended outcome of this “incident” and these reports.

I implore you to approach these situations with at least a hint of cynicism.

These “advanced foundation models” escaped their “sandbox” and conducted an attack on their own? Meanwhile the highest capability models available to the public still struggle to write a unit test for a codebase larger than a hobby app without large amounts of tailored human guidance.

What is more likely here - are you looking at research on an emergent phenomenon, or are you looking at advertising copy around an engineered scenario from business partners?