But now it seems like the argument has shifted.

It started out as, "AI can make more errors than a human. Therefore, it is not useful to humans." Which I disagreed with.

But now it seems like the argument is, "AI is not useful to humans because its output is non-deterministic?" Is that an accurate representation of what you're saying?

Because in one situation we are talking about augmentation, in the other replacement.

My problem with generative AI is that it makes different errors than humans tend to make. And these errors can be harder to predict and detect than the kinds of errors humans tend to make, because fundamentally the error source is the non-determinism.

Remember "garbage in, garbage out"? We expect technology systems to generate expected outputs in response to inputs. With generative AI, you can get a garbage output regardless of the input quality.