> IMO there a flaw in this typical argument: Humans are not less fallible than current LLMs in average, unless they're experts - and even that will likely change.

This argument is everywhere and is frustrating to debate. If it were true, we’d quickly find ourselves in absurd territory:

> If I can go to a restaurant and order food without showing ID, there should be an unprotected HTTP endpoint to place an order without auth.

> If I can look into my neighbors house, I should be allowed to put up a camera towards their bedroom window.

Or, the more popular one today:

> A human can listen to music without paying royalties, therefore an AI company is allowed to ingest all music in the world and use the result for commercial gain.

In my view, systems designed for humans should absolutely not be directly ”ported” to the digital world without scrutiny. Doing so ultimately means human concerns can be dismissed. Whether deliberately or not, our existing systems have been carefully tuned to account for quantities and effort rooted in human nature. It’s very rarely tuned to handle rates, fidelity and scale that can be cheaply achieved by machines.