As is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.

In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.

▲

geraneum 4 hours ago | parent | next [-]

The study was updated twice with the last time on August 2026.

▲

nilkn 3 hours ago | parent [-]

The data is still mostly 2025 and the 2026 data is mostly from the first few months, and most production deployments (at least in, say, the Fortune 500) lag the frontier by at least several more months (or much more). I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.

▲

simoncion 2 hours ago | parent | next [-]

> I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.

If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.

I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.

	▲	nilkn 2 hours ago \| parent [-]
		> I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now. Given my own conclusion that it's not possible for conventional academic research to accurately snapshot industrial adoption of this technology, I would expect this pattern to continue for a while until either academia becomes much faster or technology progress slows down (or both). This just isn't yet a situation where you can rely on published papers telling you what to think -- you still need to mix in a bit of your own instinct and experience and extrapolation to get an accurate up-to-date read, even if there's some prediction error as a result.

▲

geraneum 3 hours ago | parent | prev [-]

Do you have any references for the models deployed in any of these organizations?

> I suspect it's just not really possible for academic research

But we were told everything changed in fall-winter of 2025 and that when we got “real capable” agents/models. Whatever happened to that?

▲

nilkn 3 hours ago | parent [-]

I have my own experience to go off of, but this can also be inferred easily. There are many large corporations still working on "digital transformation" from 10+ years ago. There's no way such companies have fully onboarded and integrated frontier AI agents throughout their workforces with high adoption. The paper itself reports low adoption even after conditioning on access.

Everything did change after late 2025, but there are tiers of usefulness. Late 2025/early 2026 was the first time agents became clearly useful for personal projects. That's distinct from immediately having economically valuable end-to-end autonomous workflows in teams of median talent, which many would suggest is the threshold we've just recently crossed.

▲

geraneum 3 hours ago | parent [-]

If you want to refute a study which presents numbers and statistics, you have to provide it in the form of numbers and statistics.

▲

nilkn 2 hours ago | parent [-]

I don't, actually. I can simply point out that its data is out of date, and then the numbers themselves are called into question without other numbers to compare them to. And that's what I did: I shared two arguments in two comments that converge on the shared conclusion that the paper's data is not up to date (or, more precisely, cannot be up to date).

▲

geraneum 2 hours ago | parent [-]

> I can simply point out that its data is out of date

You can, but it doesn't make it correct or, evidently, yield a logical argument. Do you notice that your logic is circular? The date doesn't matter if the passage of time has not affected the outcome and that's basically what I questioned, which you refuted without any evidence. So the question still stands. Are there any comparable stats you have access to that change the picture?

	▲	nilkn 34 minutes ago \| parent [-]
		You're trying to get me to make a claim stronger than what I'm making. The claim I'm making is that capabilities have moved since the paper's data (countless benchmarks support this), and it's not possible to measure the end-to-end effect of that inside industrial applications with complete access, adoption, and internal connectivity at the same pace. Therefore, the study is fundamentally out of date and conclusions about the present or future cannot be drawn from it unless you assume that technological progress cannot have affected the relevant outcomes, which is a strong assumption you'd have to back up separately. Note that the paper's model ties end-to-end outcomes to generated code quality and correctness which is precisely one of the variables that continues to move. What the study actually shows is that organizations that had very little time to absorb older generations of a disruptive emerging technology experienced an indeterminate change in output (at the 12 month mark, anywhere from down 13% to up 25% -- the paper can't say where reality falls). It certainly doesn't establish any ceiling.

▲

omellet 4 hours ago | parent | prev | next [-]

They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.

▲

archagon 2 hours ago | parent | prev | next [-]

By that logic, it is not ever possible to do research on LLM-assisted work.

▲

bigstrat2003 3 hours ago | parent | prev [-]

> The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5.

What we have today is not meaningfully better than what we had a year ago, so that isn't relevant.

	▲	jefftk 2 hours ago \| parent \| next [-]
		Today I can write what I'm looking for, as if I'm writing a design doc, and the models can do the rest. A year ago it was faster for me to just write the code myself. Models today are enormously better than what we had in 2025.
	▲	skerit 3 hours ago \| parent \| prev [-]
		Lol