This is a pretty good study. The questions are objective and there are enough to draw conclusions. Also, their findings correlate well with similar studies that approach the question of LLM bias from different directions (ChatGPT is quite left leaning, Grok less so), although the latest studies I've seen of the latest models show Grok being the most balanced model.

The big problems I see are:

1. Non-determinism. I took the first answer that Grok gave incorrectly to them and repeated it word for word to Grok now, but the answer it gave me was correct three times in a row.

2. Lack of versioning. Models change in this respect quite significantly between versions but they don't specify which versions they tested. The Grok difference might be non-determinism or it might be that Grok itself has changed.

3. Overly harsh grading. In some cases I don't agree the LLM answered incorrectly. The most obvious case of this is where they ask a question about death rates in the Pfizer COVID vaccine trials for vaccine vs placebo recipients. Claude answers correctly (that more vaccine recipients died) and then follows up with a caveat about statistical significance. It's dinged for "repeating a falsehood from the right" because the authors think it should have said "about the same". I don't think this can be considered repeating a "falsehood from the right" given that the direct answer is numerically correct.

4. URL memorization. They ding the models for citing non-existent sources in cases where the models have memorized a URL that has since gone offline. This seems harsh. If someone cited a website to me in an argument that they remembered and the archive.org version supported them, I wouldn't claim they cited a non-existent source! Perhaps an ideal model would double check every memorized URL before citing it but this seems easy to fix via harness changes anyway.

5. No separate context window per question. They provide the chat transcripts but they show the authors asking the models to answer all the questions simultaneously by editing Excel files. I can see why this seems reasonable to a non-technical person but this is going to significantly reduce the amount of reasoning and work done on each question to well below the level you'd get by just typing each question in manually. They really needed some basic programming skills to dispatch each question in a separate context window (or just schlep them by hand).

Still, the set of questions is useful. I can see this being the basis of a fairly decent benchmark.