Is there a specific list of changes they made to the system prompt? They're claiming they removed 80% of it. That's quite substantial. It would be good to know what the model knows to do by training and what we need to avoid over-specifying in our system prompts.

Saying that "give Claude judgment" is too vague for agent implementors. Given the lack of specific details, my takeaway is that we need to go and review all context and rework prompts from prompts/descriptions from scratch until they pass the evals again.

It definitely makes me uneasy given the types of behaviours we've been hearing about from these models. Letting them use their judgement can go horribly wrong once a misaligned behaviour is triggered. It loosely translates into a relaxation of guardrails.

I worry that the ability of the model to reach similar benchmark scores to Fable is more to do with this "letting the agent off the hook", allowing it to explore a wider (but riskier) set of avenues to solve the problem than it is due to it getting genuinely better at the direct problem solving.