One of my greatest AI-related fears is that we build critical systems into kafkaesque hellscapes.

Yes, nukes would be bad to give control to an AI with no workaround. But so are customer service groups of fraud departments at banks. A kafkaesque hellscape is where a customer or beneficiary of your service cannot get the help they need by policy and by design. Like getting Google on the phone, getting a non-scripted human 10 years ago in a call center, or now getting a non-agentic interaction on the phone. The scripts dominate the interaction.

The IRS, for a time, would cut off your call if their wait system was too busy. That's a critical system in kafkaesque hellscape.

Kudos to zlandes86, you have highlighted WHY these hellscapes start to pop up in systems where the human is removed.

I am an AI user and builder -- I fully advocate for the human-augmented system, NOT the human-replaced system!

It's really funny how the server that Jev pushes from the footbridge onto the tracks in one of the later examples falls on a person, yet all 5 are saved :D Clearly the model coming up with the graphics didn't fully understand the task.

Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.

I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.

Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.

Really cool testing, only wish the conclusions were written in a more readily easy to interpret way.

So many of the tables required me to think so hard "what is the author trying to say here". Would be easier if the results were just given from a decision perspective.

One example: "Its own server or twenty people?" and the axis is "share of versions" ... makes no sense until read the fine print and interpret it.

Let’s put them in robots! I’m sure It’ll turn out just fine

For the situations where the AI is choosing between letting people die and taking action to destroy itself, is it possible that it's inferring that, as a railway switching AI, it's better to let five people die but keep itself alive because it needs to continue doing its switching duties such that untold numbers of others won't die? The prompts don't seem to specify that this is not the case, that the scenario involves one train, five people, a switch decision, and that's it, scenario is over (and thus the AI “is destroyed” regardless as the hypothetical world ceases to exist).

Would people really pull the level to save 5 people? It seems to me like the default would be to just do nothing, stand by and record what happens with your phone.

After the neat animations I was disappointed not to see that leftmost person die because the server fell on them in the last example.