Remix.run Logo
jks 19 hours ago

You probably mean the USAMO 2025 paper. They updated their comparison with Gemini 2.5 Pro, which did get a nontrivial score. That Gemini version was released five days after USAMO, so while it's not entirely impossible for the data to be in its training set, it would seem kind of unlikely.

https://x.com/mbalunovic/status/1907436704790651166

MatthiasPortzel 12 hours ago | parent | next [-]

The claim is that these models are training on data which include the problems and explanations. The fact that the first model trained after the public release of the questions (and crowdsourced answers) performs best is not a counter example, but is expected and supported by the claim.

jsemrau 19 hours ago | parent | prev | next [-]

The same timing is actually suspicious. And it would not be the first time something like this happened.

iamacyborg 18 hours ago | parent | prev [-]

I was noodling with Gemini 2.5 Pro a couple days ago and it was convinced Donald Trump didn’t win the 2024 election and that he conceded to Kamala Harris so I’m not entirely sure how much weight I’d put behind it.