I'm not sure what to make of this graph[0]. It shows medium as the most effective thinking mode by far for frontier code.

It's the only case that I saw going through the system card where more reasoning effort meaningfully negatively impacted the resulting eval. I know sometimes max efforts show a small dip, but this is substantial. I wonder why in the world that is?

[0] https://imgur.com/a/Nv8V7Ry

▲

underyx 3 hours ago | parent | next [-]

I'm not sure about the answer here, but this can be caused by the scoring rubric used by given benchmarks. For instance, if a benchmark docks scores for running too many commands or using too much wall-clock time, higher efforts will get lower scores.

▲

4 hours ago | parent | prev | next [-]

[deleted]

▲

mbil 2 hours ago | parent | prev | next [-]

Maybe it's akin to the Ballmer peak: improved performance at a specific level of relaxation

▲

km144 4 hours ago | parent | prev | next [-]

I agree, very odd they did not comment on any theories for the degradation here. Dip and then rebound at max effort is pretty interesting too. Overthinking is bad, but you can overthink so much it starts to be better again?

▲

2001zhaozhao 3 hours ago | parent | prev | next [-]

It is probably from randomness. The benchmark tasks nowadays are so long that you can't really afford to run a large number of samples of them per model & effort combination

	▲	artemisart 2 hours ago \| parent [-]
		No it's a mean of 5 runs. > We report FrontierCode’s overall score, a composite measure that grades each patch on blocking functional criteria (held-out unit tests) together with weighted code-quality rubric criteria, as mean@5. They don't explain more in the system card, I guess higher effort levels could loose points on the code quality / scope / style / maintainability stuff?

▲

landrew_ 4 hours ago | parent | prev [-]

apparently it got docked points for editing files out of scope

	▲	steve_adams_86 2 hours ago \| parent \| next [-]
		This must not be weighted very heavily on the benchmark because if it was, Opus would bomb every test (half kidding)
	▲	Dibes 3 hours ago \| parent \| prev [-]
		Do you have a source for this? That would explain it, but could be a bit of a concern on the general focus the model at higher thinking exhibits.