I don't think it's a bad way to benchmark new models, I just find it concerning that the author implies that "pelican on a bicycle" has been exhausted. At the risk of making overly broad, unfalsifiable claims I think multi-year exposure to AI content has dramatically raised our expectations for speed and volume but lowered them for quality. We see a very janky pelican and declare the problem solved.

I don't think he's claiming it's been exhausted. It's just that things have progressed to a point where people are arguing over the finer points of which pelican looks better -- which is largely a matter of taste, and an indication that we've hit the knee in benchmark where models are no longer failing in obviously awful ways.

100%. Why waste the tokens to render Lord of the Rings when the pelican test still clearly benchmarks so well.