I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task

There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.

  +-------------+-----------+-----------+-----------+-----------+--------+
  | Harness     | API total | Input     | Cached    | Uncached  | Output |
  +-------------+-----------+-----------+-----------+-----------+--------+
  | smol        |   172,807 |   142,334 |     8,704 |   133,630 | 30,473 |
  | Pi          |   427,211 |   392,767 |   137,216 |   255,551 | 34,444 |
  | OpenCode    | 1,564,429 | 1,523,957 | 1,204,736 |   319,221 | 40,472 |
  | Codex       | 3,005,744 | 2,953,154 | 2,649,344 |   303,810 | 52,590 |
  | Hermes      | 3,856,611 | 3,808,231 | 3,167,232 |   640,999 | 48,380 |
  | Claude Code | 5,073,137 | 5,029,969 | 4,587,008 |   442,961 | 43,168 |
  +-------------+-----------+-----------+-----------+-----------+--------+

I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.

Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell

Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?

The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.

> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?

> The context window is very important.

It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.

Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).

You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.

What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?

I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.

Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.

Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.

Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.

While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.

Anything else I can read about smol?

also, are you using a tool to collect those metrics? what is it?

> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.

this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.

the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.

Same task? How did the results compare?

[dead]