> on top of all that, they hope to be able to rely on "Please review the outputs" which obviously isn't an actual solution here, of course people will get complacent and throw stuff over the wall whenever they can.

This is honestly the fundamental problem of AI as I see it

When we offload our work to a different person we can calibrate our expectations to our past experiences with that person. With AI the experience is not very consistent. To use AI effectively you basically should treat it as a low trust, brand new coworker every single time you use it

That doesn't really scale, so people have two choices: be constantly hyper vigilant for mistakes the AI makes, or become complacent and trust it more than they should

People rightly point out that humans make mistakes too, not just AI. But humans have a pretty manageable cap on the amount of output they can produce. One human can pretty thoroughly review the outputs of a small team of other humans

One human can't possibly thoroughly review the volume of output that an LLM they are prompting can produce

Yeah, it's like declaring self driving safe because people are told to remain alert with their hands on the wheel, ready to take over in an instant. It's a charade.

That’s why most people that says LLM doesn’t work. It’s not that it can’t produce a good output once in a while, it’s that you can’t guarantee it. Or reduce the risks of a bad output. It’s a chaotic element and the cost of being alert enough to ensure consistency (if it’s feasible at all) is higher than just doing without.

But AI proponents are more than happier to brandish carefully curated anecdotes than to do a systematic study of risks and impacts.