| ▲ | shepherdjerred 4 hours ago | |
Am I reading the Agents’ Last Exam v1 graph correctly where it's saying GPT Luna Medium performs better than High, but XHigh is better than Medium? | ||
| ▲ | tedsanders 3 hours ago | parent | next [-] | |
The error bars on eval graphs are commonly like ±5%. It's fairly common to get out-of-order points when the gap between them is only a few percentage points. If we ran it again, or ran it with minor alterations, I'd expect the ordering to be expected. In general, we don't massage any of our eval numbers and for good statistical hygiene we'll report the first run we do, even if later runs show slightly smoother curves. (I work at OpenAI.) | ||
| ▲ | ttul 4 hours ago | parent | prev [-] | |
That is correct. The gpt-6 models are strange. | ||