I've been trying various harnesses like Pi, OpenCode, Qwen Code, and Nanocoder. A common problem I keep running into is failed tool calls, regardless of the model. What is the best harness and on-device model combination right now?

▲

arjie 31 minutes ago | parent | next [-]

I have just re-analysed most common failed tool-calls and adjusted the tool so that it works. I have a manual repair step on failure that programmatically attempts to fix some things. On failure, the harness reports the error, the repaired function, and the result. Overall, seems to work fine. But it's very model-specific. Most commonly the model fails on shell commands where it hallucinates some programs. If it does it often enough, I just promote those to commands in the PATH. Over time, it has happened less.

▲

wolttam 3 hours ago | parent | prev | next [-]

> and on-device model combination right now

That would depend entirely on what your device is. This sounds likely not to be an issue with the harness, but the capabilities of the models you've tried.

I experience almost no tool call failure using my nothing-special harness and DSv4 Flash.

	▲	docheinestages 2 hours ago \| parent [-]
		I'm looking for something that runs on an M5 Macbook Pro with 48 GB of unified memory.

▲

arcanemachiner 3 hours ago | parent | prev | next [-]

You can't afford the best model. What are your specs and what models + quants have you tried?

Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp

	▲	docheinestages an hour ago \| parent [-]
		I'm on a 48 GB M5 Macbook Pro. I use 4-bit quants with a context window of 16-32k. I tried Qwen 3.6 27B, but I can only get around 10 tokens per second, but it's painfully slow, and it often fails during `write_file` tool calls, even with Qwen Code.

▲

yearolinuxdsktp an hour ago | parent | prev [-]

Pi.dev requires some plugins to work well. Using Qwen3.6-27B/35B locally at Q8, I was quite frustrated with failed tool calls and tried many things.

Ultimately this combo worked:

1. https://pi.dev/packages/pi-tool-guard —- corrects key name synonyms and common structure errors, so tool calls succeed automatically (e.g if the model hallucinates old_str instead of oldText). It also wraps top level oldText/newText in an edits array if the tool didn’t do it.

2. https://pi.dev/packages/@aboutlo/pi-smart-edit - white-space-tolerant edits, as Qwen would sometimes add a fifth space to a four space indent

Hashline edit tools didn’t work well for me at all, they confused the model and it still failed to edit correctly. Also line removals would invalidate the rest of the file requiring re-reads. I tried pi-hashline-edit-pro, though I see it now keeps a database of hashes to help keep them stable across edits. Regardless Qwen kept thinking that the hashline prefixes were part of the source.