Remix.run Logo
▲ themgt 2 hours ago

Over the 22 harnesses, the rank correlation of the steps per pass and the cost per pass was +0.51 (p = 0.015). For the other three factors, it was −0.03 to +0.02 (p ≥ 0.88). Within each model, tool calls tracked cost as well as steps. Table 4 below and Table 5 in the appendix give the values.

For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.

Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!

▲Edward40 2 hours ago | parent [-]

Cutting the system prompt and tools is easy. Cutting steps requires understanding the model's RL strategy and how it was trained to explore.

▲bisonbear 2 hours ago | parent [-]

I wonder if Claude Code would do better given they just introduced code mode, which should significantly cut the number of steps required