| ▲ | An Empirical Study of Harness Design for Coding Agents(arxiv.org) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 115 points by wek 2 hours ago | 15 comments | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | gps372 2 hours ago | parent | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
Haven't gone through full PDF as its very detailed, few things have resonated with me so far. Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc. You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model). Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | vblanco an hour ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | embedding-shape 2 hours ago | parent | prev | next [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
The conclusions: > Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models. > Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types. > context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets > planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long > structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though: > Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default. I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | themgt an hour ago | parent | prev [-] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back" Anthropic appears to agree frontier models don't need in-session planning tools. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||