|
| ▲ | pinko 7 hours ago | parent | next [-] |
| From https://lastexam.ai/: "The dataset consists of 2,500 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting." [emphasis mine] While the private questions don't seem to be included in the performance results, HLE will presumably flag any LLM that appears to have gamed its scores based on the differential performance on the private questions. Since they haven't yet, I think the scores are relatively trustworthy. |
| |
| ▲ | panarky 6 hours ago | parent | next [-] | | The jump in ARC-AGI and MathArena suggests Google has solved the data scarcity problem for reasoning, maybe with synthetic data self-play?? This was the primary bottleneck preventing models from tackling novel scientific problems they haven't seen before. If Gemini 3 Pro has transcended "reading the internet" (knowledge saturation), and made huge progress in "thinking about the internet" (reasoning scaling), then this is a really big deal. | |
| ▲ | largbae 4 hours ago | parent | prev | next [-] | | How do they hold back questions in practice though? These are hosted models. To ask the question is to reveal it to the model team. | | | |
| ▲ | UltraSane 4 hours ago | parent | prev | next [-] | | You have to trust that the LLM provider isn't copying the questions when Humanities Last Exam runs the test. | |
| ▲ | rvnx 5 hours ago | parent | prev [-] | | Seems difficult to believe, considering the number of people who prepare this dataset, who also work(ed) or hold shares in Google or OpenAI, etc. |
|
|
| ▲ | lubujackson 2 hours ago | parent | prev | next [-] |
| I don't think any of these companies are that reductive and short-sighted to try to game the system. However, Goodhart's Law comes into play. I am sure they have their own metrics that arr much more detailed than these benchmarks, but the fact remains LLMs will be tuned according to elements that are deterministically measurable. |
|
| ▲ | stego-tech 9 hours ago | parent | prev | next [-] |
| This. A lot of boosters point to benchmarks as justification of their claims, but any gamer who spent time in the benchmark trenches will know full well that vendors game known tests for better scores, and that said scores aren’t necessarily indicative of superior performance. There’s not a doubt in my mind that AI companies are doing the same. |
|
| ▲ | Feuilles_Mortes 9 hours ago | parent | prev | next [-] |
| shouldn't we expect that all of the companies are doing this optimization, though? so, back to level playing field. |
|
| ▲ | eldenring 7 hours ago | parent | prev [-] |
| Its the other way around too, HLE questions were selected adversarially to reduce the scores. I'd guess even if the questions were never released, and new training data was introduced, the scores would improve. |
