I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.

This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.

Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics: - tweaks few decode settings like slightly higher temperature - detects when model gets stuck and tells it to "go on" - detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"

And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.

▲

sanderjd 3 hours ago | parent | next [-]

Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.

▲

Roark66 an hour ago | parent | next [-]

It is useful to compare like for like.

Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.

I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doesn't need it - or it's already done behind the scenes.

	▲	sanderjd an hour ago \| parent [-]
		In my view, the useful like for like comparison is to the entire system that people actually use. Nobody uses a "naked model", so what is the point of these benchmarks that use them in that way? I think the benchmarks should be trying to use realistic harnesses for both proprietary and open weight models.

▲

timschmidt 2 hours ago | parent | prev [-]

Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.

▲

sanderjd an hour ago | parent [-]

I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the model, then there should be some other detectable improvement, like to speed or cost.

	▲	timschmidt 42 minutes ago \| parent [-]
		I don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses. It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.

▲

kouteiheika 2 hours ago | parent | prev [-]

> Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics

Do you have a link/paper for this you could share?

	▲	Roark66 an hour ago \| parent [-]
		I haven't written one. You can easily replicate it if you wish just based on my comment and a day spent with Claude Code. In fact that is how I got the idea. There is a 4 month old post on SWEbench github that claimed 20 point boost (beating opus as well) for another small Qwen model when it's traffic was sent via a similar proxy and claude code was used as the harness. The author didn't post their full code, but he did post all sorts of logs showing which features of his proxy were activated during which run. I wanted to exclude the harness and validate his claims regarding only the proxy's impact so I reimplemplemented a subset of his features that seemed to give me the biggest "wins" with the smallest amount of work. These are the ones I list in the parent. It took one afternoon last week plus the runs that took few days. I saw about 10% better results. As I mentioned, it is a subset of tasks, not the entire set, but results so far are very encouraging. Since then I implemented many more features in the proxy (best of N for example), but i haven't had time to measure the impact yet. His proxy also had best of N, but that feature wasn't activated in on the runs he reports.