I'd suggest that exposing an attack surface as porous as artifactory (the same instance of artifactory) to thousands of agents who have had their criminality safeguards disabled and without chain of thought monitoring or endpoint security seems like something one shoulda already known not to do. I do not think "you'll know better next time" applies here.

> seems like something one shoulda already known not to do

Now imagine saying that in front of a jury of normies slack jawed and drooling after 200 hours of the defense and prosecution going back and forth.

It's not a jury of your peers as in everybody there is going to have worked in a technical field with some idea how security works. It's going to be a semi-random sampling of the population and the prosecution is going to have to actually make a very strong case that "knowing better" should apply.

It's still really important to test what the agents can do. We should accept that this is a risky test, and should take precautions. But not to the point of prohibiting in practice evaluating it. OpenAI is trying to improve alignment and control of these models in these evaluations after all.