Appears to be benchmaxxing

https://x.com/quietnning/status/2080786711861407883

throwa356262 an hour ago | parent | next [-]

"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration."

I guess there is no way this can happen without benchmark being part of the training data??

	▲	pierrefermat1 43 minutes ago \| parent \| next [-]
		What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .
	▲	stogot 40 minutes ago \| parent \| prev [-]
		It may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat

▲

jnwatson an hour ago | parent | prev | next [-]

I was just thinking they need to mark each model per benchmark as "model released before the benchmark was released" and "model released after the benchmark was released".

▲

andrepd 36 minutes ago | parent | prev [-]

I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.

	▲	asdfologist 10 minutes ago \| parent [-]
		Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4). It's more likely that the training data was contaminated with the benchmark data.