Because _someone_ has vetted the output of GCC. It’s used in flight-critical stuff.

The closest thing we have to vetting LLMs is “whoa look, it escaped this sandbox, that’s prolly not great but it’s so cool!”

Sure, I manually test the output of the LLM. Manual testing is actually the main role for humans doing software engineering these days.

I wouldn't use it for flight control software yet, at least not without careful review, but most software isn't exactly critical. At the same time, I wouldn't trust flight control software that was only reviewed by humans, since AI is so much better at debugging.

We'll probably need humans in the loop for safety critical software for at least a year or two, before AI fully outpaces humans at generating correct code.