People who are saying they're not seeing productivity boost, can you please share where is it failing?

Because, I am terrified by the output I am getting while working on huge legacy codebases, it works. I described one of my workflow changes here: https://news.ycombinator.com/item?id=47271168 but in general compared to old way of working I am saving half of the steps consistently, whether its researching the codebase, or integrating new things, or even making fixes. I have stopped writing code, occasionally I jump into the changes proposed by LLM and make manual edits if it is feasible, otherwise I revert changes and ask it to generate again but based on my learnings from the past rejected output

I am terrified about what's coming

The companies laying off people have no vision. My company is a successful not for profit and we are hiring like crazy. It’s not a software company, but we have always effectively unlimited work. Why would anyone downsize because work is getting done faster? Just do more work, get more done, get better than the competition, get better at delivering your vision. We put profits back in the community and actually make life better for people. What a crazy fucking concept right?

I find LLMs are good at essentially boilerplate code. It's clear what to do and it needs to be typed in. Or areas where I really have no idea where to start, because I'm not familiar with the codebase.

I find anything else, I spend more time coaxing them into doing 85% of what I need that I'm better off doing it myself.

So they're not useless but there's only so many times in a week that I need a function to pretty-print a table in some fashion. And the code they write on anything more complex than a snippet is usually written poorly enough that it's a write-once-never-touch-again situation. If the code needs to be solid, maintainable, testable, correct (and these are kind of minimal requirements in my book) then LLMs make little impact on my productivity.

They're still an improvement on Google and Stack exchange, but again - only gets you so far.

YMMV

I have an app which is fairly popular. This release cycle I used Claude Code and codex to implement all the changes / features. It definitely let me move much quicker than before.

However now that it's in the beta stage the amount of issues and bugs is insane. I reviewed a lot of the code that went in as well. I suspect the bug fixing stage is going to take longer than the initial implementation. There are so many issues and my mental model of the codebase has severely degraded.

It was an interesting experiment but I don't think I would do it again this way.

The last time I tried AI, I tested it with a stopwatch.

The group used feature flags...

    if (a) {
       // new code
    } else {
       // old code
    }

    void testOff() {
       disableFlag(a);
       // test it still works
    }
    
    void testOn() {
        enableFlag(a);
        // test it still works
    }

I can do one flag in ~3minutes. Code edit, pr prepped and sent.

The AI can do one in 10mins, but I couldn't look away. It kept trying to use find/grep to search through a huge repo to find symbols (instead of the MCP service).

Then it ignored instructions and didn't clean up one or the other test, left unused fields or parameters and generally made a mess.

Finally, I needed to review and fix the results, taking another 3-5 minutes, with no guarantee that it compiled.

At that point, a task that takes me 3 minutes has taken me 15.

Sure, it made code changes, and felt "cool", but it cost the company 5x the cost of not using the AI (before considering the token cost).

Even worse, the CI/CD system couldn't keep up the my individual velocity of cleaning these up, using an automated tool? Yeah, not going to be pleasant.

However, I need to try again, everyone's saying there was a step change in December.

Because its failure rate is too high. Beyond boilerplate code and CRUD apps, if I let AI run freely on the projects I maintain, I spend more time fixing its changes than if I just did it myself. It hallucinates functionally, it designs itself into corners, it does not follow my instructions, it writes too much code for simple features.

It’s fine at replacing what stack overflow did nearly a decade ago, but that isn’t really an improvement from my baseline.

AI dramatically increases velocity. But is velocity productivity? Productivity relative to which scope: you, the team, the department, the company?

The question is really, velocity _of what_?

I got this from a HN comment. It really hit for me because the default mentality for engineers is to build. The more you build the better. That's not "wrong" but in a business setting it is very much necessary but not sufficient. And so whenever we think about productivity, impact, velocity, whatever measure of output, the real question is _of what_? More code? More product surface area? That was never really the problem. In fact it makes life worse majority of the time.

Terrifying doesn't quite say it. The situation we're in is either we achieve a post-scarcity society or we'll all die.

Something I've been thinking about lately is if there is value in understanding the systems we produce and if we expected to?

If I can just vibe and shrug when someone asks why production is down globally then I'm sure the amount of features I can push out increases, but if I am still expected to understand and fix the systems I generate, I'm not convinced it's actually faster to vibe and then try to understand what's going on rather than thinking and writing.

In my experience the more I delegate to AI, the less I understand the results. The "slowness and thinking" might just be a feature not a bug, at times I feel that AI was simply the final straw that finally gave the nudge to lower standards.

It's failing when there is no data in the training set, and there are no patterns to replicate in the existing code base.

I can give you many, many examples of where it failed for me:

1. Efficient implementation of Union-Find: complete garbage result 2. Spark pipelines: mostly garbage 3. Fuzzer for testing something: half success, non-replicateable ("creative") part was garbage. 4. Confidential Computing (niche): complete garbage if starting from scratch, good at extracting existing abstractions and replicating existing code.

Where it succeeds: 1. SQL queries 2. Following more precise descriptions of what to do 3. Replicating existing code patterns

The pattern is very clear. Novel things, things that require deeper domain knowledge, coming up with the to-be-replicated patterns themselves, problems with little data don't work. Everything else works.

I believe the reason why there is a big split in the reception is because senior engineers work on problems that don't have existing solutions - LLMs are terrible at those. What they are missing is that the software and the methodology must be modified in order to make the LLM work. There are methodical ways to do this, but this shift in the industry is still in baby shoes, and we don't yet have a shared understanding of what this methodology is.

Personally I have very strong opinions on how this should be done. But I'm urging everyone to start thinking about it, perhaps even going as far as quitting if this isn't something people can pursue at their current job. The carnage is coming:/

I work as an ML engineer/researcher. When I implement a change in an experiment it usually takes at least an hour to get the results. I can use this time to implement a different experiment. Doesn't matter if I do it by hand or if I let an agent do it for me, I have enough time. Code isn't the bottleneck.

I also heard an opinion that since writing code is cheap, people implement things that have no economic value without really thinking it through.

> People who are saying they're not seeing productivity boost, can you please share where is it failing?

At review time.

There are simply too many software industries that can't delegate both authorship _and_ review to non-humans because the maintenance/use of such software, especially in libraries and backwards-compat-concerning environments, cannot justify an "ends justifies the means" approach (yet).

I don’t think the objections are not necessarily in terms of lack of productivity although my personal experience is not that of massive productivity increases. The fact that you are producing code much faster is likely just to push the bottleneck somewhere else. Software value cycles are long and complicated. What if you run into an issue in 5 years the LLM fails to diagnose or fix due to complex system interactions? How often would that happen? Would it be feasible to just generate the whole thing anew matching functionality precisely? Are you making the right architecture choices from the perspective of what the preferred modus operandi of an llm is in 5 years? We don’t know. The more experienced folks tend to be conservative as they have experienced how badly things can age. Maybe this time it’ll be different?

"it works" is a very low standard when it comes to software engineering. Why are we not holding AI generated code to the same standards as we hold our peers during code reviews?

I have never heard anyone say "it works" as a positive thing when reviewing code..

Yes, there is a productivity boost but you can't tell me there is no decrease in quality

I work in commercials.

We can now make 1$ million dollar commercials with 100,000$ or less. So a 90% reduction in costs - if we use AI.

The issue is they don’t look great. AI isn’t that great at some key details.

But the agencies are really trying to push for it.

They think this is the way back to the big flashy commercials of old. Budgets are lower than ever, and shrinking.

Big issue here is really the misunderstanding of cause - budgets are lower, because advertising has changed in general (TV is less and less important ) and a lot of studies showed that advertising is actually not all that effective.

So they are grabbing onto a lifeboat. But I’m worried there’s no land.

I’ve planned my exit.

> I have stopped writing code, occasionally I jump into the changes proposed by LLM and make manual edits if it is feasible, otherwise I revert changes and ask it to generate again but based on my learnings from the past rejected output

Isn't it a very inefficient way to learn things? Like, normally, you would learn how things work and then write the code, refining your knowledge while you are writing. Now you don't learn anything in advance, and only do so reluctantly when things break? In the end there is a codebase that no one knows how it works.

I asked opus 4.6 how to administer an A/B test when data is sparse. My options are to look at conversion rate, look at revenue per customer, or something else. I will get about 10-20k samples, less than that will add to cart, less than that will begin checkout, and even less than that will convert. Opus says I should look at revenue per customers. I don't know the right answer, but I know it is not to look at revenue per customers -- that will have high variance due to outlier customers who put in a large order. To be fair, I do use opus frequently, and it often gives good enough answers. But you do have to be suspicious of its responses for important decisions.

Edit: Ha, and the report claims it's relatively good at business and finance...

Edit 2: After discussion in this thread, I went back to opus and asked it to link to articles about how to handle non-normally distributed data, and it actually did link to some useful articles, and an online calculator that I believe works for my data. So I'll eat some humble pie and say my initial take was at least partially wrong here. At the same time, it was important to know the correct question to ask, and honestly if it wasn't for this thread I'm not sure I would have gotten there.

I don't want to generalize from my specific situation too much, but I want to offer an anecdote from my neck of the woods. On my personal sub, I agree it is kinda crazy the kind of projects I can get into now with little to no prior knowledge.

On the other hand, our corporate AI is.. not great atm. It was briefly kinda decent and then suddenly it kinda degraded. Worst case is, no one is communicating with us so we don't know what was changed. It is possible companies are already trying to 'optimize'.

I know it is not exactly what you are asking. You are saying capability is there, but I am personally starting to see a crack in corporate willingness to spend.

Basically it tricks you into making the code less maintainable, so while it seems to boost productivity, it's just delaying huge failures.

Sometimes I’m scared.

Sometimes I realise that this particular task has been slower than if I’d done it myself when I take in to account full wall clock time.

I can’t tell what type of task is going to work ahead of time yet.

  People who are saying they're not seeing productivity boost, can you please share where is it failing?

I'm going to guess that many people on HN are also on the "free ChatGPT isn't that good at programming" train.

When it comes to novel work, LLMs become "fast typers" for me and little more. They accelerate testing phases but that's it. The bar for novelty isn't very high either - "make this specific system scale in a way that others won't" isn't a thing an LLM can ever do on its own, though it can be an aid.

LLMs also are quite bad for security. They can find simple bugs, but they don't find the really interesting ones that leverage "gap between mental model and implementation" or "combination of features and bugs" etc, which is where most of the interesting security work is imo.

Same. Whenever an article like this one pops up the comments seem to be filled with confirmation bias. People who don't see a productivity boost agree with the article.

I work at tech company just outside of big tech and I feel fairly confident that we won't have a need for the amount of developers we currently have within 3-4 years.

The bottleneck right now is reviewing and I think it's just a matter of time before our leadership removes the requirement for human code reviews (I am already seeing signs of this ("Maybe for code behind feature flags we don't need code reviews?").

Whenever there's an incident, there is a pagerduty trigger to an agent looking at the metrics, logs, software component graphs, and gives you an hypothesis on what the incident is due to. When I push a branch with test failures, I get one-click buttons in my PR to append commits fixing those tests failures (i.e. an agent analyses the code, the jira ticket, the tests, etc. and suggests a fix for the tests failing). We have a Slack agent we can ping in trivial feature requests (or bugs) in our support channels.

The agents are being integrated at every step. And it's not like the agents will stop improving. The difference between GPT3.5 and Opus 4.6 is so massive. So what will the models look like in 5 years from now?

We're cooked and the easiest way to tell someone works at a company who hasn't come very far in their AI journey is that they're not worried.

I feel like this might be heavily dependent on both your task and the AI you're using? What language do you code in and what AI do you use? And are your tasks pretty typical/boilerplate-y with prior art to go off of, or novel/at-the-edge-of-tech?

It’s been my experience as of recently. I point it at an issue tracker and ask it to investigate, write a test to reproduce the problem and plan a fix together. There’s lots of hand holding from me but it saves me a lot of work and I’ve been surprised by its comfort with legacy code bases. For now I feel empowered, and I’m actually working more intensively, but I was wondering to myself if I’m going run out of work this year. Interestingly, our metrics show that output is slowed by increased workload on reviewers.

I think you can more stuff done earlier but the quality is not good or it doesn't work as expected if you tinker with it enough. Fixing the issues from the generated code usually doesn't work at all

I'm with you. The project I'm working on is moving at phenomenal velocity. I'm basically spending my time writing specs and performing code reviews. As long as my code review comments and design docs are clear I get a secure, scalable, and resilient system.

Tests were always important, but now they are the gatekeepers to velocity.

Outside of coding/non-physical areas, the impact can be quite muted. I haven't seen much impact on surgical procedures, for example (but maybe others have?).

I’m currently working across like 5 projects (was 4 last week but you know how it is). I now do more in days than others might in a week.

Yesterday a colleague didn’t quite manage to implement a loading container with a Vue directive instead of DOM hacks, it was easier for me to just throw AI at the problem and produced a working and tested solution and developer docs than to have a similarly long meeting and have them iterate for hours.

Then I got back to training a CNN to recognize crops from space (ploughing and mowing will need to be estimated alongside inference, since no markers in training data but can look at BSI changes for example), deployed a new version of an Ollama/OpenAI/Anthropic proxy that can work with AWS Bedrock and updated the docs site instructions, deployed a new app that will have a standup bot and on-demand AI code review (LiteLLM and Django) and am working on codegen to migrate some Oracle forms that have been stagnating otherwise.

It’s not funny how overworked I am and sure I still have to babysit parallel Claude Code sessions and sometimes test things manually and write out changes, but this is a completely different work compared to two or three years ago.

Maybe the problem spaces I’m dealing with are nothing novel, but I assume most devs are like that - and I’d be surprised at people’s productivity not increasing.

When people nag in meetings about needing to change something in a codebase, or not knowing how to implement something and its value add, I’ll often have something working shortly after the meeting is over (due to starting during it).

Instead of sending adding Vitest to the backlog graveyard, I had it integrated and running in one or two evenings with about 1200 tests (and fixed some bugs). Instead of talking about hypothetical Oxlint and Oxfmt performance improvements, I had both benchmarked against ESLint and Prettier within the hour.

Same for making server config changes with Ansible that I previously didn’t due to additional friction - it is mostly just gone (as long as I allow some free time planned in case things vet fucked up and I need to fix them).

Edit: oh and in my free time I built a Whisper + VLM + LLM pipeline based on OpenVINO so that I can feed it hours long stream VODs and get an EDL cut to desired length that I can then import in DaVinci Resolve and work on video editing after the first basic editing prepass is done (also PyScene detect and some audio alignment to prevent bad cuts). And then I integrated it with subscription Claude Code, not just LiteLLM and cloud providers with per-token costs for the actual cuts making part (scene description and audio transcriptions stay local since those don't need a complex LLM, but can use cloud for cuts).

Oh and I'm moving from my Contabo VPSes to running stuff inside of a Hetzner Server Auction server that now has Proxmox and VMs in that, except this time around I'm moving over to Ansible for managing it instead of manual scripts as well, and also I'm migrating over from Docker Swarm to regular Docker Compose + Tailscale networks (maybe Headscale later) and also using more upstream containers where needed instead of trying to build all of mine myself, since storage isn't a problem and consistency isn't that important. At the same time I also migrated from Drone CI to Woodpecker CI and from Nexus to Gitea Packages, since I'm already using Gitea and since Nexus is a maintenance burden.

If this becomes the new “normal” in regards to everyone’s productivity though, there will be an insane amount of burnout and devaluation of work.

I see an individual productivity boost, but not necessarily a collective one.

I don't think features per hour is really what is holding back most established businesses.

My experiences suggest that we still have some time before the people that understand the plumbing of the business _and_ AI bubble up to positions of authority through wielding it practically and successfully at increasingly greater scale.

A terminology tangent because it's an econ publication: Notice that the article doesn't talk about productivity.

Productivity is a term of art in economics and means you generate more units of output (for example per person, per input, per wages paid) but doesn't take quality or otherwise desireability into account. It's best suited for commodities and industrial outputs (and maybe slop?).

> I am terrified about what's coming

Why? This is great. AI fixing up huge legacy codebases is just taking the jobs humans would never want to do.

Meanwhile gemini tells me my go code doesn't compile (it does)

Gaslight me by telling me I must be a time traveler because I use go 1.26 but the latest version actually is 1.24

And tell me I can't use wg.Go() because this function does not exist (it does)

> my job is easier now, I do less. > I am terrified about what's coming.

God I hope I never ever have to work with you

I can only explain it by people not having used Agentic tools and or only having tried it 9 months ago for a day before giving up or having such strict coding style preferences they burn time adjusting generated code to their preferences and blaming the AI even though they’re non-functional changes and they didn’t bother to encode them into rules.

The productivity gains are blatantly obvious at this point. Even in large distributed code bases. From jr to senior engineer.

You were probably deficient in RESEARCH skills before. No offense to you, since I was also like this once. LLMs research and put the results on the plate. Yes, for people who were deficient in research skills, I can see 2-3x improvements.

Note1: I have "expert" level research skills. But LLMs still help me in research, but the boost is probably 1.2x max. But

Note2: By research, I mean googling, github search, forum search, etc. And quickly testing using jsfiddle/codepen, etc.