| ▲ | Edward40 2 hours ago | |||||||
Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task frontier-harness-eval@v1.1.0 dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours. For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens. https://hub.harborframework.com/datasets/frontier-harness-ev... | ||||||||
| ▲ | seizethecheese 2 hours ago | parent [-] | |||||||
This doesn't answer the question at all. | ||||||||
| ||||||||