Yes: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Although you can certainly do a better-and-worse job of preventing these kinds of issues.

Direct access to the database, and create the "refund program" on the fly. Yes, stuff of nightmares.

At some level everything an agent does is through a "programmatic interface" (tool calls).

Some people might use skill-based scripts, MCPs, or some kind of raw access to a database. My point is that well designed CLIs are the optimal programmatic interface, for many reasons.

> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.

I believe this is the general belief about basically every human skill, that if you stop doing the technical fundamentals you get worse at understanding the activity. The question is whether coding is like sailing a square-rigged wooden ship, which became completely useless knowledge after the invention of the steam engine, or if it's like playing an instrument, which while technically unnecessary after the advent of MIDI and other tools, absolutely hurts your ability to arrange, compose and perform if the skill is neglected.

For my money: I think the AI scenario is more like the latter, but "humans are worse at coding" isn't the consequence I see coming. I worry that in ten years we will be awash in software that's impossible to understand. I don't think that's happened in any human industry ever. Someone has always understood how the machines are built, even if they're very remote from the users of the machine.

No serious programmer is regularly bottlenecked by typing speed. Even the ones who type slowly.

If you find yourself writing repetitive code you should consider adding a layer of abstraction. If your language isn't powerful enough you can write a code generator.

> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.

Like, perhaps, understanding that it is free of security and functionality bugs.

This is such a delusional take it's borderline trolling. Code is an expression tool to precisely describe a process that does useful thing. Typing prompts is not too different from writing some very vague code, which is arguably a waste of time by itself.

The typing was never the bottleneck.

> a live, running process that does useful things

That is one of the things code does. It also communicates the developer's thoughts about how that process should work to others. If the latter is neglected, the code becomes very difficult to collaborate on. Very few lines of code that are written are "write once". Mostly they're changed, repeatedly, over time by many people. The live, running process is a very temporary entity by comparison. Yes, it needs to exist and do useful work. No, it is absolutely not the only thing that matters.

> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.

I would argue that this is nearly always the case. I don't think people really understand programs that they've only read at more than a very superficial level. This is why I tend to make (temporary) small changes, printlns, etc. when exploring a new code base: it aids greatly in understanding how a program actually works.

And it's even worse (in my experience) with LLM generated code, as it tends not to result in particularly understandable code. It is a lot like LLM generated prose: it often looks entirely reasonable at a surface level, but has a of weirdness/incorrectness hidden beneath the surface. But that surface level makes it very hard to avoid glossing over the details when reviewing the code. For this reason, I personally find it's much more effort to carefully review code than it is to write it.

Humans make mistakes all the time, but their code tends to naturally be structured for human understanding (to some degree based on skill/experience) because they themselves needed to understand it to write it.

I think LLMs are very useful tools, but after quite a lot of experience using them, I think it's generally better to use them as a sounding board, or to help you get unstuck or remove points of friction. Using them to write all of your code (at least for me) seems like a net negative.

I also think it's extremely easy to overestimate how much time they save. It feels like they're a productivity boost because it takes less intense focus to implement something. But I've experienced several instances where actually writing the code myself would have been both quicker and have resulted in better code.

All that being said, it can also be really hard to not write all of your code with agents once you get used to it. There's also a kind of slot-machine-like effect where you write a prompt, excited for the result, and when it doesn't quite come out right, you think "ah just one more prompt and it'll be good." It's hard to see when you're actually doing it though.

It's also weird to me how much people think typing is what the LLM is replacing. Typing was never the hard part. It's the translation of the high-level idea into an unambiguous process that's hard. That's also the valuable part, that requires thinking through the edge cases and consequences of decisions, and that just gets glossed over when using an LLM unless you rigorously review what the LLM has done.

At the end of the day there's a real tradeoff to be made, and it's worth being conscious of what's being given up.

The behavior of an LLM is not and cannot be “well-specified”.

>rather than giving it in a well-specified and tested form

So, code?

Because the LLM version will have countless number of bugs and security holes, which means you will spend weeks or months of your life fixing them.

This is a truth that many are having a hard time accepting. Getting shoved into the light so fast is blinding.

Because it’s rarely so black and white. Knowing the inputs and outputs is merely the first steps, you need to think about the transitions too as they have their own costs.

Those costs don’t disappear and it’s truly naive to think they don’t matter. Take security issues, they may arise because what you thinks was the input is merely a subset of the true input range. And the extra possibilities lead to unforeseen behavior.

A lot of programming is about ensuring that the input and the output are the sets defined in the specs. And the rest is that the transition/relation is the right tradeoffs of performance, correctness, and costs.

...and now think about Claude being shutdown by the gov... :-D

Every single person who bootstrapped becoming powerful did it by rushing into things, but it's a high variance strategy because you could also end up destitute

Use 'AI' to create anti-AI reels showing how much they suck at all tasks. Spam CEO's underlings.

In every case up until now, a jump in abstraction has moved us forwards in ease of understanding the underlying artifact at a conceptual, human level. High level languages are effectively runnable documentation.

It's possible to say that LLMs producing code may be the same category of thing, but the non-determinism and ephemerality of it all makes it difficult to imagine.

If my experience having to manually unf*ck “production” slop written by sweatshop tier offshore told to go wild with a Claude subscription is any indication then we are a LONG way off from any of that BS.

My job is safe because I’m the only person at the company that actually understands what the actual code is doing and I’m the one that gets the calls at 2am and weekends.

“Weird LLM bytecode”

Why not just generate object code for the target mschine directly?