That actually would be significantly better, because then you could work out under what conditions they were wrong and correct for it. But being stochastically wrong means you have to check everything, because you never know when the errors will creep in.

Compilers have a ton of hard optimization passes, and many of those are solved well enough via stochastic algorithms. Nobody actually fully optimizes register allocation or whatever, and a few random restarts can go a long way toward some better performance.

It seems like your points are:

1. Programmers sometimes do something other than what they're told, so what's the harm in letting an LLM be disobedient instead?

2. Managers sometimes don't ask for the right things, so why not let an LLM figure things out?

To both points, those observations are literally the reason why we have software engineers writing code instead of management. The job of fully understanding the implications of all requirement interactions in excruciating detail is not a management responsibility. In the context of using LLMs as a higher-level compiler, I don't think people are arguing that management should be the people writing that higher-level code.

Separately, the failure modes are very different. If I ask a SOTA agent and a SOTA harness to construct a minimal sandbox for executing subprocesses without network access using OS sandboxing primitives to ensure you don't have Scun*horpe-esque ways of sneaking networked commands through (and assume I expanded that into a proper prompt instead of the high-level summary I'm using to convey the idea here), there's a nontrivial chance it implements some of the sandbox by string matching the commands instead of using proper OS primitives. It'll have thousands of tests showing it works, and it'll still be garbage even just from a requirements point of view and ignoring code quality or everything else we're being told to ignore when using them as higher-level compilers. There's also a nontrivial chance that if you ask either the same or a new session to check on that property it will conclude that the sandbox was implemented correctly. Good human engineers will either get the job done or tell you that they can't for some reason or another. That "looks like it succeeded but actually failed hard and you can't tell unless you read the code" error case is new to LLMs. Yeah, we write bugs from time to time (as do LLMs), but complete failures to understand the core product so badly that fixing the "bug" requires a full rewrite, one likely to introduce its own similarly catastrophic rewrite-inducing bugs, are a different beast altogether.

And again, I use LLMs a lot, almost exclusively, but they have limitations, and failing to recognize those isn't conducive to success. The frontier keeps moving, so you might find these current observations irrelevant eventually. People are choosing to not read the code today though, and that's bad.