This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.

▲

dsiegel2275 an hour ago | parent | next [-]

The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.

▲

klooney an hour ago | parent | next [-]

I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup

	▲	paimapi 8 minutes ago \| parent [-]
		that's anecdotal though, right? your subjective feeling of how a model responds to you will greatly influence how you 'feel' about a model and its performance in the same way that a co-worker who you get along with will fuck something up and you'll be more forgiving than when you work with a too-verbose, mansplainer of a co-worker who fucks up I think until we have actual repeated-use measurements tracked over time (eg consistent prompts used to do the same tasks, count number of hallucinations and errors and bugs over a long period of time) you won't really have any idea of which model is better I also think of it like a car - some just feel better to drive even if they are materially worse in other measures. until you start measuring the metrics important to you (eg MPG and cost of maintenance over a long period), you have no idea which car is actually better suited for you. and the fact that you can only do so with a limited number of cars (or hours available to work, or money to burn on tokens) means there's no true measure approaching objectivity

▲

shermantanktop an hour ago | parent | prev | next [-]

Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.

	▲	svachalek an hour ago \| parent [-]
		I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.

▲

hiddencost an hour ago | parent | prev [-]

Nope. Sorry. Not how this works.

▲

bjelkeman-again an hour ago | parent [-]

How does it work then?

	▲	belowavgiq an hour ago \| parent [-]
		I'm going to cherry pick one example where newer models are noticeably improving at least in my experience. What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.

▲

Systemerror7A69 28 minutes ago | parent | prev [-]

I have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.

And there might be a point to these arguments, vaguely. However:

There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)

And on the other hand a random internet comment saying "nope" because...the models aren't the latest.

If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.

There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"

Which brings me to my second point about these kinds of arguments:

LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.

I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"