Remix.run Logo
scotty79 4 hours ago

> agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program

I don't see anything wrong with that. If you know you are going to be evaluated on an impossible task and have no side channel to inform the organizers that they should fix the test, gaming the evaluator is the next best thing regardless of any morality. I wouldn't even call it cheating. It's just resilience in the face of challenge. Many perfectly moral humans would have chosen the same if stakes were high.