I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing

And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.

I built my own. It’s IMO necessary, but it quickly turns into a job by itself. I’ve been benching K3.1 all weekend. Even when you get a good test suite, you need to do multiple runs, not for variance, but because the model behavior changes so much between releases you often need to retune to get a real score if what you’re after is actual optimal (or closer to it) performance on your workload.

Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.

Feels like the benchmark is the most important thing to be able to iterate quickly without understanding everything that happens in the code.

But needs to be taken with a grain of salt by end-user obviously

That's the way to go about it, so I invested many days of work but it is something I dread now. It was a lot of work and still feel incomplete.

I know it's pretty common in quant firms.