> Warning: AI benchmarks. Please don't trust them

I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.

I think the point is we need better and open benchmark players which can bring and monetise on domain specific benchmarks for end users and not just selling them to the big labs.

That’s true, for example here, this benchmark chooses tasks and datasets that Jeffy is really good at, better than Jev or Laya: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...

But the Jeffy models are explicitly trained on those datasets (to be fair, classifiers are just really good, even a decision model finetuned on the same dataset, doesn’t beat a classifier - in classification tasks)

I generally trust benchmarks but my personal experience in the last 6-9 months is they have little relation to real LLM performance, unfortunately.

I've had better luck just running a few of my own simple tests.

100% agree, have found through tests a big discrepancy between models optimised for SWE benchmarks and usage in prod - there is a somewhat subjective element to it as well in terms of output that can't be quantified effectively (less so with decision models naturally)

I like to test the decision models with coin flips. This has a well known answer and I haven't seen one give it back. It undermines my trust in every other decision.

Or, you know, it's an easy excuse because they're already behind on many of them.

[dead]