Remix.run Logo
▲ bisonbear 3 hours ago

These are cool results and I'm glad that tokens are being invested to eval harness capabilities.

However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.

I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)

https://www.stet.sh/blog/cost-per-successful-task

▲Edward40 2 hours ago | parent [-]

"Cost per pass" is distinct from "cost per successful task."

Cost per pass is simply the average cost per task divided by the pass rate.

If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.

▲bisonbear 2 hours ago | parent [-]

In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.

To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode

▲Edward40 an hour ago | parent [-]

We use cost per pass, not average cost per task, because we want to see the harness keep a high pass rate and low cost at the same time.