Sadly, this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology, "don't be evil", etc. And Google is run by managers now.

Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.

>> this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology

This is not true. It is needless negative hyperbole that IMHO doesn't belong on HN.

While this might be overly cynical, it's also a powerful validation of the exact feedback loop that you want to have in an organization.

If what you say is true, then the next set of benchmarks will be written to illuminate the true reflection of real-world tasks. Then those get benchmaxxed so managers can show "numbers go up."

Repeat enough times, and you get materially and genuinely improved products.

The burden is on the press, outside researchers, and general tech industry to both write and care about the metrics that matter.