I've never been fond of the argument that there should be a professional software engineer certification, but hearing people like you being presented with the potential dangers and just going 'oh yeah just go with a better test suite and you can just wing it' makes me seriously reconsider.

Vibe code administrative systems for your local golf club to your hearts desire for all I care, god forbid somebody will have to stand around a bit longer before going for their 9 holes. But safety critical equipment is not the place to fuck around with the code prediction machines that have existed for 4 years, have been writing more-or-less acceptable code for 2, and will still regularly refer to themselves as MechaHitler or just make up shit. "Yes you're absolutely correct, I was wrong" doesn't help you one bit if you have just been chewed up by heavy machinery, and the fact that people like you exist who go 'oh just a few more more unit tests surely will fix it' is a terrifying thought.

But don't humans make mistakes too? Like are we sure the failure rate of AI with the right checks and bounds is lower than humans, who are flawed machines themselves?

If you need assurances, have a different LLM write the test suite.