Verification.

That’s the whole issue in a nutshell.

Can the output of a generative system be verified as accurate by a human (or ultimately verified by a human)

Experts who can look at an output and verify if it is valid are the people who can use this.

For anyone else it’s simply an act of faith, not skill.

Agreed. There are other skills in play too though, such as knowing how to narrow the problem space to increase the chance of a good response.

It would be great if responses were tagged with uncertainty estimates.

Generative systems don’t really reduce the need for expertise, they just change its role. And yeah, without verification, you’re not coding with AI - you’re gambling with it.

This is the crux of why I think AI code is a waste of time

It is much more difficult and time consuming to build a mental model of AI generated code and verify it than to build the damn thing yourself and verify it while it is fresh in your memory