Remix.run Logo
▲ seizethecheese 3 hours ago

Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!

A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.

▲Edward40 2 hours ago | parent | next [-]

Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task frontier-harness-eval@v1.1.0 dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.

For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.

https://hub.harborframework.com/datasets/frontier-harness-ev...

▲seizethecheese 2 hours ago | parent [-]

This doesn't answer the question at all.

▲dotancohen 2 hours ago | parent [-]

It's still the most informative thing to come out of that account or the linked article.

▲jzemeocala 2 hours ago | parent | prev | next [-]

For real, i remember setting up advanced Linux configurations back before the 3.0 kernel and I'm pretty sure I had a higher failure rate.

The biggest difference is that I, as a human, get mentally exhausted. but modern AI (when given unlimited tokens and tools) is like if Stephen hawking had a meth habit and the tenacity of Sisyphus with a bad case of syphilis after being told he can be cured if he can get that boulder over the hill

And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"

▲dotancohen an hour ago | parent [-]

That quote is outdated. My phone nags me now twice daily to update, and the update removes features I use so I'm careful to tread around it.

My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.

Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.

We've definitely passed the point where the derivative has gone negative.

▲Edward40 2 hours ago | parent | prev | next [-]

I'm also building a leaderboard in the meantime, which will release in a few days with more detailed harness version info and methodology.

▲seizethecheese 2 hours ago | parent | prev [-]

Confirmed. This is 17 TB4 tasks and 45 TB2.1 tasks. Why?

▲Edward40 2 hours ago | parent [-]

We want to see how performance diverges between TB2.1 and TB4 while keeping costs manageable, since running the entire dataset is really expensive.

▲seizethecheese 2 hours ago | parent [-]

How were the specific questions chosen?

▲Edward40 an hour ago | parent [-]

You can see the task in https://hub.harborframework.com/datasets/frontier-harness-ev...

We kept the category unified so we have tasks for every domain like ML , science and others.

▲seizethecheese 38 minutes ago | parent [-]

How were the specific questions chosen?

▲Edward40 a minute ago | parent [-]

[flagged]