Remix.run Logo
andrepd an hour ago

I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.

asdfologist an hour ago | parent [-]

Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4).

It's more likely that the training data was contaminated with the benchmark data.

mupuff1234 44 minutes ago | parent [-]

You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check?

They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .