Remix.run Logo
w-m 10 hours ago

Interesting tool, congrats on the launch!

I was wondering: have you thought about automation bias or automation complacency [0]? Sticking with the drop-tables example: if you have an agent that works quite well, the human in the loop will nearly always approve the task. The human will then learn over time that the agent "can be trusted", and will stop reviewing the pings carefully. Hitting the "approve" button will become somewhat automated by the human, and the risky tasks won't be caught by the human anymore.

[0]: https://en.wikipedia.org/wiki/Automation_bias

dhorthy 9 hours ago | parent | next [-]

this is fascinating and resonates with me on a deep level. I'm surprised I haven't stumbled across this yet.

I think we have this problem with all AI systems, e.g. I have let cursor write wrong code from time to time and don't review it at the level I should...we need to solve that for every area of AI. Not a new problem but definitely about to get way more serious

exhaze 8 hours ago | parent [-]

This is something we frequently saw at Uber. I would say it's the same as there's already an established pattern for this for any sort of destructive action.

Intriguingly, it's rather similar to what we see with LLMs - you want to really activate the person's attention rather than have them go off on autopilot; in this case, probably have them type something quite distinct in order to confirm it, to turn their brain on. Of course, you likely want to figure out some mechanism/heuristics, perhaps by determining the cost of a mistake, and using that to set the proper level of approval scrutiny: light (just click), heavy (have to double confirm via some attention-activating user action).

Finally, a third approach would be to make the action undoable - like in many applications (Uber Eats, Gmail, etc.), you can do something but it defers doing it, giving you a chance to undo it. However, I think that causes people more stress, so it’s rather better to just not do that than to confirm and then have the option to undo. It’s better to be very deliberate about what’s a soft confirm and what’s a hard confirm, optimizing for the human in this case by providing them the right balance of high certainty and low stress.

dhorthy 8 hours ago | parent [-]

i never thought about undoable actions but I love that workflow in tools like superhuman. I will chat w/ some customers about this idea.

I also like that idea of:

not just a button but like 'I'm $PERSON and I approve this action' or type out 'Signed-off by' style semantics

foota 7 hours ago | parent [-]

I think the canonical sort of approach here is to make them confirm what they're doing. When you delete a GitHub repo for example, you have to type the name of the repo (even though the UI knows what repo you're trying to delete).

If the table name is SuperImportantTable, you might gloss over that, but if you have to type that out to confirm you're more likely to think about it.

I think the "meat space" equivalent of this is pointing and calling: https://en.m.wikipedia.org/wiki/Pointing_and_calling (famously used by Japanese train operators)

dhorthy 6 hours ago | parent [-]

this is cool. I have been an andon cord guy forever

j45 7 hours ago | parent | prev [-]

Premature optimization, and premature automation cause a lot of issues, and overlooking a lot of insight.

By just doing something manually 10-100 times, and collecting feedback, both understanding of the problem, possible solutions/specifications can evolve orders of magnitude better.

dhorthy 7 hours ago | parent [-]

yeah the people who reach for tools/automation before doing it themself at least 3-10 times drive me crazy.

I think uncle bob or martin fowler said "don't buy a JIRA until you've done it with post-its for 3 months and you know exactly what workflow is best for your team"

j45 5 hours ago | parent [-]

I am starting to call that Harry Potter AI prompting.

Coding with English (prompting) is often most useful where existing ways of coding (an excel formula) can’t touch.

Using llms to evaluate things like an excel formulas instead of using excel doesn’t feel in the spirit of using this ai’s power.