The perverse incentives created by these AI leaderboards are crazy.

The leaderboards are dumb, but I understand the point of telling people not to worry about tokens and just use it. They are trying to get people to try it, to discover new uses without asking “is this worth testing”. It’s basically early R&D budget. Eventually these companies will decide it’s time to transition into efficient usage.

But we need OKRs rocks and METRICS! Everyone must have their own one numberrrrr!