| ▲ | Edward40 2 hours ago | |||||||
"Cost per pass" is distinct from "cost per successful task." Cost per pass is simply the average cost per task divided by the pass rate. If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line. | ||||||||
| ▲ | bisonbear 2 hours ago | parent [-] | |||||||
In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail. To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode | ||||||||
| ||||||||