For personal projects, I defer almost everything day-to-day to the models, so it gives me a good feel for what the models are good at.

Even then, it's kind of a wash these days between the sota models, and we're talking about maybe a 10% performance difference or something. But every once in a while there's the experience of one model spinning its wheels on a bug/repro/issue while another model comes in and one-shots the solution.

That make sense. Let me ask you this When the project reaches a level of complexity . Do you simply reach for better models or do you reengineer it or does the project scope stop at the egdge of the model's capabilities.

I am asking because in my personal projects after a while they becomes a giant messy ball of wires and i basically trust the model to untangle it for me , by the time it untangles properly, I run into my token limits.

In general, many of these stories of "one one shots it and the other spins on it" is down to trajectory divergence and can be achieved simply using another instance of the same model.

While just simply trying many times independently gets you the improvement that is due to pass@k vs 1, you can get huge improvements if on top of that, depending on your setting, you find a way to ensure some stochasticity by perturbing tool calls, etc and running multiple instances.

The general theme is, embrace the stochasticity rather than the leaky abstraction on top of it.

With modern LLMs, investing in this kind of harness tooling is much more fruitful than hoping for the best from the model.

While many basic instances of this are built in to the popular harnesses (much of cursors higher-than-usual success rate with older models was due to really excellent context mgmt), you can never beat one that is optimised for your particular codebase, infra and general setup.

Until last year or so, the context management needed varied too much at too coarse a level across different models and even model instances, but now they are all extremely robust in a much higher % of contexts and are thus way more amenable to developing context management tools for, without needing to do a research teams worth of evals.

Custom evals and harnesses are thus extremely high ROI now. We are finding companies needing to do less and less tweaks and getting much fewer regressions (you should have reg tests in ur evals) with every new usecase and every new model.

It can be really simple to start with: change your grep/rg that it uses to a script that does in effect "rg $@ | shuf".

More complex examples are: giving different subagents different tools, randomly failing tool calls, truncating file reads randomly, having a small model invent N possible failure modes causing a bug and appending that to N prompts and starting subagents from each - this all forces each to pursue different paths. $example_specific_to_your_company_setup is highest ROI though, since most companies actual failure modes are dominated by idiosyncratic API shapes and retrieval quirks that no usual harness will bother modelling.

Also important IMO to not assign any meaning or semantically interpret the CoT as an acceptance mechanism (it is ok to use it as a rejection mechanism e.g if you see it plotting a sandbox escape whether it eventually emits the exploit or not is not something you want to hedge). We have to resist the temptation and ensure we only interpret tool calls, codegen, etc in our evals and only think of the cot as "some output that pushes the conditional distribution" which may or may not semantically match the typical preceding tokens of the desired tool call.