The mainstay benchmarks are becoming a farce and not partially relevant to what customers actually care about.

Metrics like price per million tokens are meaningless when the models are wildly inconsistent and unpredictable on how many tokens they use to complete a task.

The labs all need to move to variable pricing so they don’t go bankrupt, but customers won’t accept a world where nobody can predict what things will cost. It’s becoming an unavoidable problem.

At times it also feels like the labs actually encourage these models to burn useless tokens as they are incredibly verbose unless you really push them to not be. If you just ask something simple that could get a 5 word response you get a whole useless essay.