| ▲ | root_axis 3 hours ago | |
Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on. Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output. | ||
| ▲ | jwpapi 2 hours ago | parent [-] | |
Also controlling input | ||