> Or how do you even know it's the wrong answer if you didn't think it through before starting?

Real life is full of unknown unknowns, AI risks putting you on the wrong path without you even realizing it.

It doesn't help a bit that (almost? I'd say all) AI products are very confident while they're guiding you on that unknown unknown path, so of course that you don't realize soon enough the epistemological situation said AI product has driven you into.