Before LLMs, I've watched in horror as colleagues immediately copy-paste-ran Stack Overflow solutions in terminal, without even reading them.

LLM agents are basically the same, except now everyone is doing it. They copy-paste-run lots of code without meaningfully reviewing it.

My fear is that some colleagues are getting more skilled at prompting but less skilled at coding and writing. And the prompting skills may not generalize much outside of certain LLMs.

Just the fact that even retail quality hardware is still improving at local LLM significantly is still a great sign. If AI quality remained the same, and the cost for local hardware dropped to $1000, it would still be the greatest thing since the internet IMO. So even if the worst happens and all progress stops, I'm still very happy with what we got.

The overall trend in AI performance will still be up and to the right like everything else in computing over the past 50 years, improvement doesn't have to be linear

And how many years of experience you needed to know what to write, and what if you can replace that time with how long prompting takes?

You're anthropomorphizing.

LLMs can't lie nor can they tell the truth. These concepts just don't apply to them.

They also cannot tell you what they were "thinking" when they wrote a piece of code. If you "ask" them what they were thinking, you just get a plausible response, not the "intention" that may or may not have existed in some abstract form in some layer when the system selected tokens*. That information is gone at that point and the LLM has no means to turn that information into something a human could understand anyways. They simply do not have what in a human might be called metacognition. For now. There's lots of ongoing experimental research in this direction though.

Chances are that when you ask an LLM about their output, you'll get the response of either someone who now recognized an issue with their work, or the likeness of someone who believes they did great work and is now defending it. Obviously this is based on the work itself being fed back through the context window, which will inform the response, and thus it may not be entirely useless, but... this is all very far removed from what a conscious being might explain about their thoughts.

The closest you can currently get to this is reading the "reasoning" tokens, though even those are just some selected system output that is then fed back to inform later output. There's nothing stopping the system from "reasoning" that it should say A, but then outputting B. Example: https://i.imgur.com/e8PX84Z.png

* One might say that the LLM itself always considers every possible token and assigns weights to them, so there wouldn't even be a single chain of thought in the first place. More like... every possible "thought" at the same time at varying intensities.

Human code is still easier to review. Also, I program 80% of the time and review PRs 20% of the time. With AI, that becomes: I review 80% of the time, and write markdown and wait 20% of the time.

Yeah, I also highly value consistency in my projects, which forces me to keep an eye on the LLM and steer it often. This limits my overall velocity especially on larger features. But I'm still much faster with the agent. Recent example, https://github.com/igor47/csheet/pull/68 -- this took me a couple of hours pairing with Claude code, which is insane give the size of the work here. Though this PR creates a bunch of tables, routes, services -- it's not just greenfield CRUD work. We're figuring out how to model a complicated domain, integrating with existing code, thinking through complex integrations including with LLMs at run time. Claude is writing almost all the code, I'm just steering

then have ai write a deterministic transformation tool that turns it into the specific style and standard that is needed

As well as the peer comment about Gladwell (10k hours is considered the point you've mastered a skill), it's also a far more honest metric about how much time you've spent actually programming.

Maybe you were writing code, make design choices and debugging 8 hours a day. Maybe you were primarily doing something else and only writing code for an hour a day. Who would be the better programmer? The first guy with one year of experience or the second guy with 7 years?

I personally would only measure my experience in years, because it's approaching 3 decades full-time in industry (plus an additional decade of cutting my teeth during school and university), but I can certainly see that earlier on in a career it's a useful metric in comparison to the 10,000 hours.

I think it’s probabky because of the malcom gladwell “ten thousand hours” idea.

No, I never expect AI to one-shot (if I see such a miracle, it's usually because I needed a one-liner or something really simple and well documented, which I can also write on the whiteboard from memory).

Try iterating over well known APIs where the response payloads are already gigantic JSONs, there are multiple ways to get certain data and they are all inconsistent and Claude spits out function after function, laying waste to your codebase. I found no amount of style guideline documents to resolve this issue.

I'd rather read the documentation myself and write the code by hand rather than reviewing for the umpteenth time when Claude splits these new functions between e.g. __init__.py and main.py and god knows where, mixing business logic with plumbing and transport layers as an art form. God it was atrocious during the first few months of FastMCP.