This is a great example of how to test how well a benchmark score generalises. The first order approximation is not complicated: just vary a parameter and see how much the score changes.

Which parameter to vary depends on the result of the test, but the MaxxBench route of varying the task spec seems like a good default, though not always the most informative.

In one of my private benchmarks I realised many of the less capable models didn't fail because they submitted the wrong answer, but because they exceeded their budget for number of tool calls they were allowed. The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).

Was my benchmark actually measuring ability to combine tool calls rather than reasoning ability on the problem at hand? Well, easy enough to find out: re-run under a different tool call budget and see how much the scores change.

(Next problem: is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark? It could be if it is a sign of planning multiple steps ahead.)

▲

ronsor a day ago | parent | next [-]

> is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark?

Yes, models are generally rewarded for this during RL. Not combining tool calls is an objective deficiency, even if not a major one.

▲

celrod a day ago | parent | next [-]

Yes, two ways failing to combine tool calls is a real deficiency for the LLM's user: it costs money and can get you rate limited.

Every turn forces you to pay $ for cached inputs and possibly gets you rate limited on TPM. At 500k context and $0.26/M-tok cache read price, every time it combines two tool calls instead of taking separate turns would save you $0.13. That could add up.

Maybe the provider sets a rate limit of 2 million TPM. That's just 4 turns in 1 minute before you start getting 429s. Cutting them in half by combining calls can save that.

All else equal, we definitely want models combining tool calls!

	▲	zhengzefeng 18 hours ago \| parent [-]
		[flagged]

▲

Luker88 20 hours ago | parent | prev [-]

It also makes models completely ungovernable and unfollowable.

Asking for user permission before an action is pointless. it's not going to use a single `write`, it's going to run a 25 line perl script without even newlines.

At this point just completely remove permissions, and leave only one tool, a rust-like scripting language.

But this also drives usage up, because you rely less on yourself, and more on the increasingly black-box that is the subagent-of-subagent that you are not even going to check the result of.

	▲	viraptor 20 hours ago \| parent [-]
		It depends on what you actually want to ask permissions for. We're still doing trivial script execution, but we could go further. Capabilities like "do you allow giving access to JIRA read only token" would be a massive improvement. But there are other paths too.

▲

verdverm 16 hours ago | parent | prev [-]

> The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).

So tool call limits during benchmaxxing is why they have gained this behavior, Goodhart strikes again...

I look forward to the near future when training is more accessible to more people and we can have more experimentation and diversity in models.