This is a good point, actually. Might well be the reason.

This seems to be the issue with using LLMs for any code generation. Even with my own code bases that I've written entirely by hand over years, if I use AI to implement anything, I don't go through the mental model of architecting it, so I don't know how it works. I can only imagine this to be far, far worse for large code bases maintained by a team of people who are all using AI.

> while you could ask your LLM to do that you aren't going to run your bank on the result.

Why not?

I feel like we're entering a new era of prejudice against not a category of humans, but against non-human intelligences.

The design patterns for distributed and fault-tolerant systems are well-known and established in the industry. Both humans and AIs are familiar with them!

So if you sketch a design for the AI to follow, establish the rules in AGENTS.md, have a robust test suite, use a frontier model dialed up to eleven, etc... why not rely on the LLM output?

At the end of the day, humans are not without fault either.

I've been wading through some legacy "pre-AI" code recently and it has more bugs than a rainforest! Static fields used incorrectly, causing data races. Floating point types used for money amounts. JavaScript and SQL injection up the wazoo. Wildly unsafe password handling. So on, and so forth. This is the norm for most human-written software, not the exception.

As a proof-of-concept, I tried an AI rewrite of one such legacy app[1], and it is not bug free, but it notably has fewer bugs than the original. Different bugs, sure, and I'll have to iron them out after a round or two of UAT, but I'm honestly more confident with what I got from the chatbot than the code inherited from humans.

[1] Deals with money, but admittedly at a much lower level of risk and consequence than a banking app running on a mainframe.

It's a clean-cut financial decision.

The existing system works. Yes, it costs a lot to maintain, and you could definitely reduce that if you moved to a more modern system. So now you're talking payback periods. Cost of development / maintenance cost savings per year = number of years before you pay back the project.

Problem is, that the cost of the development is often unclear, and the maintenance cost savings, while definitely above zero, and often unclear, and approximated the numbers usually come to a payback period in decades.

And that's without the usual tech caveats; We can't promise there won't be bugs. We can't promise deadlines will be met. We can't promise the project will succeed at all. We can't promise existing functionality will be faithfully reproduced in the new system. The normal risks around any software dev project.

All in all, it looks really expensive and really risky compared to just doing nothing and running the same old system for another five years.

Source: I helped do some of the maths on this for a Y2K project.

Pushy fundamentalists, I suppose.

what is the metaphorical blowing up here?

I have no issues recognizing that I had memory-related problems in production (I program embedded systems in C).

But most of my issues were related to concurrency and data sanification, especially when the other end of communication fails with unexpected behavior. These bugs are nastier than memory.

So, I have pointers, and I am not afraid to use them.