You should see the error bars between human programmers.

Only if you have a tiny screen.

You can make them output what you would've written yourself, but that would be slower than writing it yourself, so no one does that - but it’s trivially true.

Incidentally I’ve always thought that about working on cars, but more like skilled mechanics may still vibe and not care. Amateurs may be thorough but much slower, may need to do a lot of research before every step. Just caring does a lot.

I agree with you about “thoroughness” being key here, but I’d additionally advocate that we need to leave room for people to make some slop learn it.

Part of the learning process, whether you’re a “skilled” or “unskilled” engineer (or not an engineer at all, and noting that going from one to the other is just a matter of learning), is being able to “vibe and not care.” Yes, it’s also helpful to identify things that might not work out theoretically in advance, using the things we learn in our CS programs. But coding with LLMs is a brand new modality, and we are all indeed just learning to work with them.

I think alternating vibe coding with vibe-assisted DRYing/cleaning is ultimately a workflow enhancer that can make better software faster. But engineers have to be allowed to make some slop to learn it.

I asked an agent for some high level cleanup and refactoring. I found that I ended up disagreeing with most of the structural changes. Some of them were necessary, some were beneficial, but largely it turned straight line code into abstract factory manager type stuff.

More broadly I've found that making code more elegant (e.g. by removing duplication) increases the cognitive load, because now you can't just read the code anymore but need to mentally "decompress" the higher level structures and indirection into the straight line code, the "code that actually runs."

> LLMs can clean up after themselves if you steer them. It's not very hard. I often have a two-step dance I do where the LLM first outputs some code and then I prompt it to fix the types up to my standards.

My experience has been the opposite. I prompt it and it generates something that mostly works, but steering it into something that would actually be maintainable is an exercise in futility. GPT-5.5 uses up my entire allocation of tokens in just reading the context docs and doing a single shot task.

> aven't found a way to prompt it with any number of skills or CLAUDE.mds or anything else to get it to do it the way I want on the first pass, but it's not that hard to just fix it afterwards.

I've yet to find a way to get an LLM to actually follow the instructions in CLAUDE/Agents.md It very quickly gets to a point where it forgets explicit instructions such as "run clang-format on all .h/.cpp files that you touch", and saying "our coding standards can be found at <link to Notion/Confluence>, please ensure all code adheres strictly to this standard" is ignored. I'd also say that the first pass of the code is very often not even close to how it _should_ be implemented so it's not just review and patch up, it's rearchitect + restructure 50% of the code.

> it also rapidly starts changing things just to change things,

Agreed. LLM's are (very good) text generators. They are good for generating code, and left to their own devices they will generate and generate and generate. Getting them to edit, simplify, and foresee future problems is something that people keep saying "use the latest model, it's amazing (despite saying that about the last 3 models" or "you just need to use <harness|framework>" or "your agents.md needs to contain XYZ" will solve.

Can you elaborate on the tests thing? I'm new to testing, and AI agents recently voluntarily added an large number of tests to one of my projects. I've been learning a lot by reading them, and it seems like a great habit to develop.

But they're also writing some very strange code and some very strange tests. I don't know what good practices look like here, so when something looks strange to me I can't trust my own judgment, whether it's actually smelly or just a pattern I'm not used to yet.

--

On a side note, I recently had an agent implement a major architectural change. It turned out to have done it completely backwards, in a way that was pointless. (Improved nothing and actively made things worse.) However it had supplied generous tests for the new code, and of course all the tests passed...

So it had "proven the correctness" of something which was completely incorrect.

I later realized that even formal verification would not have prevented this. It would have just written a mathematical proof that the wrong code was correct.

You forgot to include link to your tool.

I would tell my co worker “this code has so many issues that I’m not actually reviewing it. Please make sure it meets the project standards before sending it back to me”if it happened twice I’d talk to their manager.

Can you share a transcript of an LLM generated feature/bug fix that you steered into being good quality?

you'd be amazed how many do just that and are incredibly bad at anything else that does not involve opening up an IDE that spoon feeds them everything they need.