> is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark?

Yes, models are generally rewarded for this during RL. Not combining tool calls is an objective deficiency, even if not a major one.

Yes, two ways failing to combine tool calls is a real deficiency for the LLM's user: it costs money and can get you rate limited.

Every turn forces you to pay $ for cached inputs and possibly gets you rate limited on TPM. At 500k context and $0.26/M-tok cache read price, every time it combines two tool calls instead of taking separate turns would save you $0.13. That could add up.

Maybe the provider sets a rate limit of 2 million TPM. That's just 4 turns in 1 minute before you start getting 429s. Cutting them in half by combining calls can save that.

All else equal, we definitely want models combining tool calls!

It also makes models completely ungovernable and unfollowable.

Asking for user permission before an action is pointless. it's not going to use a single `write`, it's going to run a 25 line perl script without even newlines.

At this point just completely remove permissions, and leave only one tool, a rust-like scripting language.

But this also drives usage up, because you rely less on yourself, and more on the increasingly black-box that is the subagent-of-subagent that you are not even going to check the result of.