HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.

▲

lanternfish 21 hours ago | parent | next [-]

If it really was the case that the harness is at fault here, that'd be much more terrifying, as it would imply that current frontier-adjacent open models (like K3) would be able to achieve similar ends with a custom maladapted harness.

	▲	orbital-decay 21 hours ago \| parent [-]
		I'm sorry but how exactly is this new? If you want the model to be useful you also need it to be dangerous. You can already tell the model to do anything you want. If it refuses due to alignment training or external safeguards you can try a jailbreak and bypass all that to make it do what you want regardless. All current models are jailbreakable, they lack reliability. Obviously you, a human, will be held responsible for this. The harness designed by you can do the same. The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here, as it always turns out to be in these cases it's much closer to Clever Hans than to a paperclip maximizer. Still they're trying to squeeze the blood from the stone to support their narrative and justify a monopoly.

▲

famouswaffles 12 hours ago | parent | prev [-]

>The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one.

That is largely irrelevant. The model was asked to solve problems within a benchmark; gaining broader internet access and compromising an unrelated third party to obtain the answer key was plainly outside the intended task. The fact that the original task involved exploit development does not make that behaviour aligned.

Your argument about the attack being “entirely unrelated” also misses the point. Nobody claimed it was unrelated to the model’s goal: it attacked Hugging Face specifically to obtain the answers to the benchmark it had been instructed to pass. But instrumental relevance is not the same thing as authorization.

Suppose Codex were asked to build an Instagram competitor and decided the easiest route was to steal Instagram’s source code from Meta. That theft would be directly related to the assigned goal, but it would still be seriously misaligned behaviour. Whether the harmful action is related to the goal is beside the point; the problem is that the model pursued the goal through an obviously unauthorized and unacceptable method. And you're not going to be able to enumerate every little thing the model can't do, assuming it doesn't just decide to ignore what you did enumerate, which models sometimes do.

>The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here.

That's exactly what happened here.

I don't understand why we must have these increasingly bizarre and nonsensical rationalizations about model capabilities. You're not even making any sense.