Nothing more real in research than academics fighting over credit and missed citations.

That was my initial reaction, however, there is more to it.

The thing that made these breakthroughs feel important was that they seemed to be open ended problems that the system itself made independent progress on. That the results were not primarily great retrieval into the corpus of mathematical research + stitching together.

A major point of having novel frontier problems as a benchmark itself is to try to sidestep this issue a bit, where it's hard to tell if one is evaluating reasoning vs retrieval capabilities.

But at least one of the problems was identified as definitely being plausibly mostly retrieval, since it hinges on but does not cite a 2016 paper that the author identified. Closely enough that the author describes it as plagiarism. Another problem seems to combine results from 2016 and 2019.

Ignoring the question of credit, it makes it look like OpenAI doesn't actually understand the eval well in the first place, and undercuts the notion that the results represent great leaps forward in mathematical reasoning.

▲

ofjcihen 21 hours ago | parent | prev [-]

I mean are they wrong to want credit? This isn’t like “I brought coffee to the meetups and corrected some spelling errors” but more like OAI pulling the “you made this? I made this” meme.

▲

scarmig 19 hours ago | parent [-]

You can dig through every single paper and find some citation that was missed. No one cares that much unless the result is important, and then you get bitter recriminations saying that it was stolen or plagiarized. Tale as old as time. See e.g. Schmidhuber, who has a long list of vendettas against people he thinks stole his work.

	▲	ratorx 11 hours ago \| parent [-]
		The existing, human standards are based on human capabilities. Humans may not have read all the papers that could be cited. However, all these papers are almost certainly in the AI’s training set and it potentially has the ability to retrieve them over the internet into its context. Why shouldn’t we expect the AI to cite more comprehensively? It seems more like a goal-setting problem, than an actually novel one. But a paper with lots of citations is not as “cutting edge”, so maybe doesn’t make sense for marketing reasons.