Most of us effectively function as next-token predictors, in my experience. Some of us are trained not to dereference null pointers, but there's always a non-zero probability that we do so anyway. To reduce that risk I use the same tools an LLM would, i.e. code review, automated testing and verification, etc. There isn't obviously much difference between humans and LLMs in that sense.

Like you I would not vibe code something important, but I think that's primarily because I want to have a deep understanding of a system before I accept responsibility for it. An LLM would still be useful during implementation and likely to increase the quality of the result.

It seems very likely that AIs will become capable enough that they can take complete responsibility for development, deployment and maintenance of the systems we're used to building. Further, it seems likely they'll end up much more capable than humans have been. I think that's an eventuality we need to be prepared for.

A well-utilised AI tool can probably already build a much safer and more robust security/legal/user related system than you or I can, in any given time budget. There's a question around what happens as humans are required less for the "well utilised" part, and who is responsible when humans are not really involved anymore.