One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.

Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.

The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.

Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.

I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!

This problem started before AI hit the mainstream. Make the company “more efficient” (i.e. cut costs) by outsourcing to contractors in areas with lower labor costs. Then, remove escalation paths, then remove abilities from them so that can’t do crazy things like refund a customer. Finally, automate the whole thing and be certain there’s no option available that could possibly return money to the customer.

AI has no problems. Just properly follow proper procedures.

https://www.youtube.com/watch?v=gEyFH-a-XoQ

Yeah, that's one of the points of the EU AI Act, I don't have the specifics, but for some critical sectors decision making cannot be automated by AI and you are obligated to get a human explanation if you request it.

Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.

ah yes kafkaesque. the one with the bug