From Terminal-bench-2.1 details,

> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.

This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.

For reference, in tbench-2.1,

1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)

2. 8 out of 89 tasks allow 8GB RAM

This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1

As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.

▲

rsanek 43 minutes ago | parent | next [-]

This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.

▲

kingstnap 2 hours ago | parent | prev | next [-]

Why are resource limits considered at all aside from models accidentally fork bombing themselves?

I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?

	▲	GodelNumbering 2 hours ago \| parent [-]
		Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation. A few examples from memory: 1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train). 2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs. 3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes [1] https://github.com/harbor-framework/terminal-bench-2-1/blob/... [2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...

▲

efromvt 2 hours ago | parent | prev | next [-]

Out of curiosity, how often are the resource limits the bottlenecks? What do harnesses do to help here - limit parallelism? More efficient tools?

	▲	artrockalter 2 hours ago \| parent [-]
		The task could be verifiable in the environment so limiting its CPU and RAM could be to discourage brute forcing the answer.

▲

meric_ 43 minutes ago | parent | prev [-]

Huh? What are you talking about?

https://www.anthropic.com/engineering/infrastructure-noise

Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either

▲

GodelNumbering 22 minutes ago | parent [-]

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

	▲	meric_ 15 minutes ago \| parent [-]
		Yes but my point is - Resource limits are a "recommendation" and are not strictly enforced - Significantly boosting resources up to 3 did not statistically shift performance results Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results