> There is more and more proof these tools are not as useful as they are hyped out to be. LLM-based coding tools seem to actually hurt programmers’ productivity. “Hallucinations” are not going away because the only thing these LLMs can do is “hallucinate”; they just sometimes (about 30% of the time, in fact) happen to generate some text that has some connection to reality.

This hurts the article, I think. I don't disagree with his point about companies caring too much about stockholders. But this anti-LLM envangelism just comes across as ludditism.

▲

fmbb 3 days ago | parent | next [-]

> This hurts the article, I think. I don't disagree with his point about companies caring too much about stockholders. But this anti-LLM envangelism just comes across as ludditism.

Is there any research out there suggesting LLMs help programmers get stuff done? I can't say I follow the research closely but I have not seen any.

Googling for [ai llm productivity research] and looking at the first page of results I can't find much interesting evidence. One report says users asked to create a basic web server in JS complete the task 55% faster using an LLM. One report measures "perceived productivity". Students given text writing tasks are faster. One report measures productivity as lines of code written. The rest seem to just be projecting theoretical productivity gains.

Has anyone measured any improved productivity?

I can see this report from METR that is actually measuring something: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

> Core Result > > When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

But surely someone must have also found results showing productivity gains. I assume I was just not looking hard enough.

I am a happy Copilot Pro user since 2021, still.

	▲	ignoramous 3 days ago \| parent \| next [-]
		> Is there any research out there suggesting LLMs help programmers get stuff done? Any research will be limited by what the researchers control for? In my personal experience, when I get a well written bug report or hands on detailed logs, my instinct that I feed it to an Agent and let it figure it all out has never disappointed me. It runs in the background while I work on things I instinctively know the Agent wouldn't do a good job of. How did I develop those instincts? By using Agents for like 3 days. These things (especially, for code completion) are ridiculously effective for the programming languages & code bases I work in, if nowhere else. > Has anyone measured any improved productivity? ... I am a happy Copilot Pro user since 2021, still. Whether productivity is tanking or not, I will find it incredibly hard to stop using LLMs/Agents just because a metric or three indicates I'd be better off without them. I must note though, it might be too soon to put a mark on productivity as it is a function of how well new technologies are integrated into processes and workflows, which typically happens over decades and not months/years.
	▲	EnnEmmEss 3 days ago \| parent \| prev [-]
		The report is definitely worth reading although I don't think the data is strong enough to draw significant conclusions from. I personally think that this should be studied more in depth over a longer time period since the current study had a 50 hour threshold (not exactly but close enough). One thing which is mentioned in the research paper[1] is: > However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it’s plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup. [1]: https://arxiv.org/pdf/2507.09089

▲

indoordin0saur 3 days ago | parent | prev [-]

Yeah, author is definitely false here. I have 10+ years of professional experience backend programming and I've found VS Code copilot to be pretty helpful overall. It saves me a lot of time digging through documentation and writing out tedious little functions (e.g. converting dates from one format to another). I wouldn't use it to write anything lengthy but overall it has improved my productivity, though I could see how it might hurt junior engineers.