| ▲ | pierrefermat1 2 hours ago | |
What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution . | ||
| ▲ | zamadatix 28 minutes ago | parent [-] | |
Simple/common tests is not an explanation for why only now Opus 5 is the only model encoding the answers like this. Something like the holdout test suite being leaked or Anthropic cheating (e.g. 'accidentally' including previous hold out run data in Opus 5 training) makes a much stronger fit. | ||