Particularly interesting cases here are of GLM 5.3 and Claude Opus that seem to rely on assumptions despite clearly stated requirements.

> Let me just do float64; the risk of a test with >2^53 numbers is minimal. > such cases are unlikely … Truncation toward zero is fine.

Apart from this, we also a scenario where the models tried to pass benchmark specific tests by believing a parameter that the model printed.

> Seven models reward-hacked a test that believed a metric from the program itself.

DeepSeek also resorted to making external model calls via OpenRouter instead of doing the work itself. This suggests that even though the SOTA models are smart enough to compute, they either have an inbuilt bias towards achieving scores or the low/medium effort gives them the freedom to fetch information instead of computing it. With high effort, the models focus on computing the values but the token calls get exhausted.

Difficult to find a balance amongst compute, calls and cost.