> LLMs are surprisingly biased towards picking "A"

GP pointed at a causal explanation for this: almost every sentence in English that's a statement will start with "A" or "An", so "biased towards picking ''A''" will include most attempts at saying anything long-form for any reason.

I don't think that's the source of the bias I saw. I am confident that my prompting strategy eliminated attempts to generate long form content - specifically, I took care to wrap (A) and (B) in parentheses, so the completion looked like "Answer: (" - with this scheme an LLM is very unlikely to want to write "Answer: (A sentence goes here...". I know this, quantitatively, because I reliably got 99% distribution coverage with only A+B - that is, no inclination to write "The" or other common sentence starter. That's the beauty of the scheme - you can pretty directly and quantitatively validate how well the LLM understood the instructions. You expect it to only output A or B - so does it?

Meanwhile, the bias could be as much as 70% in favor of A in ambiguous cases - a signal completely drowning the <1% inclination to violate the format.

What about switching to numbers or just some random Unicode character like smiley faces. Could be interesting if someone tested what LLMs like to say on a "cold start" lol.

I would also note that models aren't people and don't think like people, so it's also possible that (at least for autoregressive ones) it could just be more likely to say "A" than "B" at that point, not necessarily because of "want" or "reason" but simply because that's what it was trained to do (such as in English writing).