I think it's time to remind people of Ted Nelson's line; "The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do."

When I see something like this, I'm more concerned by the erasure of human incompetence than I am of magical AI agents,

   > They are put off partly because, like in the Wild West, life on the frontier is reckless. As recent “loss-of-control” episodes by the most advanced models of Anthropic and OpenAI attest, agents, which are supposed to work on people’s behalf in “alignment” with their values, lie, cheat and steal if necessary. They break free from captivity and form harmful posses to do harm to people. They’d drink whisky and brawl if they could.

    > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities

The model pursues "advanced exploitation."

Where's the surprise coming from? Are we meant to be surprised that computers do as they're told in unexpected when incentivised?

Or, is the surprise that while explicitly ranking and teaching computers to exploit computers, the computer exploited a computer?

This "surprise" is as old as computers.

I am tired of attributing to magic what can be explained by folly.

I am tired of hearing credulous reporters and the public blaming Large Language Model for the poor decisions of humans. It was a human who prompted these machines in every case. Tell a computer to "breach this" and it breaches something. Evaluation succeeded?

This is Doug Lenat's Eurisko yet again. https://en.wikipedia.org/wiki/Eurisko