So basically it turns out the good models are good and the benchmaxxed models are benchmaxxed.

Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.

https://news.ycombinator.com/item?id=50406948

But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?