The spread isn't as large as I thought from the headline. It's closer to a toss-up than I would have expected.

The length of the response is a huge factor:

> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.

So the score is partially a proxy for longest responses.

Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?

Yeah it's hard to get good data on what's best. We did read the a good sample amount of essays. Lots of arguments about what made them better, but its uncontroversial that longer ones were more preferred.

Interestingly we trained a simple LORA layer on top of Inkling and it turns out a model can smell which model wrote a response 70% of the time. I wonder if the smell of gemini is just more preferred by students.

Hmm, seems like which model teachers prefer would be the more important headline…