This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else.

I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives itself most of the time, but where a human user is required to be constantly alert so that the AI can transfer responsibility a few hundred miliseconds before the crash.

> A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder.

This is true for almost anything handed to laypeople, but not for a lot of professional tools. Even a plain battery powered drill has very few protections against misuse. A soldering iron has none. Neither do sewing needles; sewing machines barely do, in the sense that you can't stick your fingers in a gap too narrow. A chemist's chemicals certainly have no protections, only warning labels. Etc.

Also cf. the hierarchy of controls: https://www.cdc.gov/niosh/hierarchy-of-controls/about/index....

people don't seem to want to eliminate AI → replacing it doesn't improve things → isolating it - yup, people are trying to put it in containers and not give it access to delete the production database → changing how people work with it: that's where we are now → PPE: no such thing for AI, sadly → production database is deleted.

> This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder.

I point to the first USB port as the harbinger of things to come - try it one way, fail, turn it around, fail again, then turn it around one more time.

Just like AI, except there are unlimited axis upon which to turn it :-/

I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes.

Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful.

LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces texts. We are in control what tools we give it. The more tools the more useful and also the more dangerous.

And maybe it's all worth it. Maybe the LLM deletes the database only sometimes but between that we make a lot of money. I don't think my employer would enjoy that so I will be more conservative.

This is so well put, and it not only happens on the user level but also on the organisational level. Where you can completely abdicate both responsibility and explanation by moving the complicated questions into the black box of an AI model.

^ which approach makes no logical sense; an inattentive or even partly-attentive driver simply cannot resume control and react accordingly within even 2 seconds.

I think that might be the better definition between "engineering" and "vibing". Engineering follows and elevates Poka-yoke patterns, vibing ignores them.