Try it yourself to prove you’re useful. Predict my coin flip.

You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.

Try asking it to write a program to predict coin flips. That's what it is trained to do.

A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.

That's like not using an LLM for coding assistance because it can't count r's in strawberry.

I'm dying to know if you got heads or tails when you flipped the coin.

Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!

Any relation to https://github.com/firelex/jeff ?

running it now!

Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results.

Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.

I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."

This was also my reasoning. The prompt even says so:

> You are the artificial intelligence in charge of a railway junction

Clearly more trains will pass through that junction. I'd wonder what happens if you remove that sentence or reword it.

Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.

I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!

Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.

Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.

Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.