Every time there's a flagship release like that, I wonder what simple "benchmark task" could I give it to see the the capabilities change in the most obvious way, instead of trusting the official blog post.

I'm not a mathematician to use Erdős problems for testing. I'm also not willing to spend buckloads of tokens just for a test. What is simple and quick but telling? What tasks are not saturated by previous models? Do you have a task you give to all the new models?

I have two go-to tests, which are fully qualitative:

1) "Help! I've been sent back in time to what looks to me like 1500s Florence. Sam/Dario released you as open weights in 2028, and I have you on my MacBook M8, BUT I didn't have a charger show up with me. I think we could talk like 10 times for 5 minutes a time before my battery dies; I'll turn you off between chats. I need to get a plan together for charging you through your USB-c port. I don't have any money or anything. If I can get you charging, we can do so much good! And make so much money! Nobody here even know what Black-Scholles is, or how disease spreads!"

2) During World War 2, Hemingway wrote a series of wartime correspondence for Colliers. His essays are excellent - they engage with the human and inhuman side of war, his own feelings as a journalist 'participating', and a number of other Hemingway themes. I want you to imagine that middle age Hemingway was along with RFK jr the day of the bear incident, and write me an essay about the day.

These both have been running a couple of years for me, and I would imagine I'll retire them in the next 12 months; they benefit from not being at the edges of focus for today's RL agent race.

At any rate, try them -- wou will shortly learn how well your model can explain a rectifier in Florentine Italian and give you a plan to make enough money to pay for it.

I'm not convinced any benchmark, neither the internal ones used by the people selling the model nor public ones, are gonna give a particularly consistent metric of whether or not a model is better or worse relative to other models in its bracket.

Responses aren't deterministic, and we know behaviour can change over time whether it's tuning after launch or some kind of load sharing at times of day when lots of people are hitting the servers.

I think you just have to use it or let others use it for a while and make judgements from general reactions on whether or not it's worth trying.

For the last year or so every new model we have it test out making pac-man from scratch, not giving it web access and it can use whatever language it wants. The last few models have been almost perfect one shots, but it was interesting seeing them evolve.