Real-world code bases are already far too large for any one programmer to internalize. They might as well be uninspectable. This problem isn't going to get anything but worse in the years to come. Incremental improvements in languages and methodologies won't suffice; future systems will simply have to be designed differently. They will be based on very high-level descriptions that can be used to synthesize tests that programs written and maintained by AI models can satisfy.

Human involvement will end after the first phase, once the required test targets exist.

Basically we are going to have to stop micromanaging our toolchains and start telling them what outcomes we want. Call it "vibe coding lol" or whatever, that's how it's going to work. When you write code in a traditional high-level language, you are telling your compiler how to generate the code that actually gets executed by the CPU. But this object code, again, might as well be completely inaccessible as far as most developers are concerned. If the compiler were to achieve the same result in different, unknowable ways each time it runs, that's a horrifying notion to those same developers... but we all might as well get used to it, because soon there will be no other way forward.

Notice that the math guys are already having to deal with similar problems. Mochizuki or somebody drops a thousand-page proof on the community, and experts at the very highest levels have to waste years looking for bugs in it. They can't go on like this and they know it. They will have to give up the quaint idea of understanding something completely in order to prove or refute it, just as programmers and engineers will have to relinquish control over how our own creations work.

I suspect that some sort of AI will progress to some point of usefulness, the fact that it can do as much as it can right now should count for something. On the other hand, these llms seem to still go off into the weeds time and again, and from what I read of folks experiences, I can't drop the deterministic mindset as easy as folks like you can. Still, it doesn't matter, like you say these things are here and everyones throwing time and money at them, so hopefully my concerns are overblown and unfounded.