> a benchmark for our expectations to see how much they've grown for new model releases?

Posters have been saying they'd be happy with open-weight Fable 5 since before F5 was public and I saw one poster on HN say it last week. And with those posts there's usually a few mentioning the truism 'goalposts tend to move' and we haven't met our future selves or seen the future context we'll be in.

Studies show humans are poor predictors of our own future wants, needs, preferences and behaviors and few are fully aware how far and often we gradually move goalposts. Some of those posters weren't really making a precise, quantified statement. Many were expressing something more directional like "Wouldn't it great if {best, most costly thing I want but can't get or afford} were {readily available free or cheap } ?" Those variables update every time the sentence is output.

And in recent months models are saturating more of the benchmarks HN posters relate to and more of the real-world workloads we have. Six month ago, model flaws were more obvious and common. They still happen today, but for more of us, they're rarer, more complex and workload dependent. It's getting harder or takes longer for us to see clear, consistent differences between similar models and model versions. You can even see it in Simon's fanciful bike-riding pelican 'not-a-benchmark'.

I'm not saying that models are now great at everything (or even close), just that for a lot of us they're good enough at our current things that it's getting harder to personally gauge by how much. So we have to rely on complex composite benchmarks (which increasingly don't reflect any one thing at high granularity) or proxy reports about 'other people's things' like frontier theoretical mathematics.