There are hundreds of benchmarks (all noisy), dozens of model families, and dozens of modes and variants within each. Announcements are crafted by publicity teams and involve cherry picking a chart that looks compelling.

As a prospective user, none of of the benchmarks details will matter to you, but the family of models and settings they peg themselves against can give you a hint of where you might want to evaluate them within your own workflows.

As an enthusiast or lay researcher, the more interesting thing is to look at their techniques and any innovations they're exploring in training, inference, etc. Those are usually going to be details tucked below the fold and won't be summarized in a benchmark chart.

As anybody else, why do you even care about this in the first place?

This doesn't just apply to AI. You should know better than to take marketing announcements at face value, nor trust that most "influencers" or "journalists" or "commenter" repeating and discussing those announcements are doing a lot of work as they buzz over what's said.

Don't be a victim to PR spin, but also don't discount potentially interesting work just because it got channeled through a marketing team. Just look for the information you need amidst the noise or move on if there's none there.

> Announcements are crafted by publicity teams and involve cherry picking a chart that looks compelling.

*crafted by LLMs with a list of PR guidelines in a markdown file