| ▲ | bisonbear 2 hours ago | |
In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail. To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode | ||
| ▲ | Edward40 43 minutes ago | parent [-] | |
We use cost per pass, not average cost per task, because we want to see the harness keep a high pass rate and low cost at the same time. | ||