As of their latest update (AA's), I find their index hard to trust, and a bit shady. 60% of the benchmarks contributing to the AA index are now private, nobody can replicate them. And they explicitly say that they offer benchmark consulting to labs. This is a serious conflict of interest.

The latest update knocked down only openweights models from the leaderboard.

I agree that they are a bit shady, but private benchmarks are the only way to prevent benchmaxxing

Can’t blame them. Silicon Valley is expensive.