For fun I've been vibe coding something I know well: toolchains. Maybe not the right thing to vibe code. But I can more or less judge the quality of the output.

When left to its own devices with the instructions "make an assembler for the architecture in ISA.md" -- well Claude picked Python as the implementation language. Tokens lifted through a bunch of regex. No expression parser! Oh dear. My first assembler was like that too, to be fair.

However, when I described the desired passes and their types:

    collectDefines :: [SourceLine] -> Either AsmError ([SourceLine], Map Text Text)
    
    runLitPool :: [SourceLine] -> Either AsmError ([SourceLine], [(Text, LitKey)])
    
    evalExpr :: Text -> Map Text Text -> Either AsmError Int

So where AI has deterministic inputs and outputs it is extremely good to the point I think that there's a theoretical issue around computational there.

Like - it can do the work for us.

It jives with post training and verifiable rewards.

The reason AI doesn't do well at 'architecture' is 1) are are bad at it and have given it a lot of mush and 2) we don't have good abstractions for it.

The result is - you stick to 'very strong conventions' and if you walk of that path you're risking a lot.

Toolchains are very deterministic, the AI can take it apart and re-assemble like Lego - and each level of the space is also deterministic. It's perfect for AI.

LLMs are bringing us back to all the “proper” software engineering stuff that we’ve always known we should be doing, but until now we never had enough time/people/money to do it right.

Brainstorming and research before writing a design.

Writing a design or spec before writing the code.

Comprehensive unit tests.

Etc etc etc.

Like you, I get vastly better output from the tool when I create a detailed spec in markdown before I let it start coding. And bonus, the LLM is pretty good at helping with the spec too.

I keep telling people that they have to design and think about it first and then go to the tool, but they keep saying “Claude can plan too” and obviously it produces some shit that requires a lot of changes while when I get it to go I can almost always one shot the stuff I want because I am actually putting in the time to give it a detailed plan of what to do.

Even just saving me the time to deal with CI is worth it.

It doesn’t even to be that complex. You can just say do comprehensive research and analysis in the space and give me an implementation plan. Then if it is 20 steps, I ask it to implement 3-5 at once. It’s essentially been one shot for everything I can throw at it.

>Code is mediocre in many places.

As if code written by devs at major corporations is't mediocre at best.

Nokia's Symbian OS took days to build. Days. With a D. Not minutes, not hours but days.

One of our devs shipped code to prod with a memory leak thanks to including a library that had "do not use this library in production because it causes a memory leak" written everywhere as warning.

So I don't wanna hear about how poor AI code is when human code is shit too. Human laziness and stupidity can beat AI hallucinations.

Sure, maybe your DeepMind, OpenAI devs and your John Carmacks of the world can beat AI code 100% of the time, but most workers most companies get don't have John Carmack as candidates.