They do disclose that they scored much lower than Fable on SWEBench Pro, which is a pretty high-quality benchmark. I think it's partially just about what they choose to emphasize...

▲

danielsamuels 2 hours ago | parent | next [-]

It's worth noting that OpenAI recently came out saying, "We don't think SWEBench Pro is worth reporting any more" - https://openai.com/index/separating-signal-from-noise-coding...

	▲	37374848 2 hours ago \| parent [-]
		lol

▲

bel8 2 hours ago | parent | prev | next [-]

SWEBench Pro should be ignored until they fix it or disprove the broken task accusations.

▲

futureshock an hour ago | parent | prev | next [-]

There has been a lot of chatter ever since the Mythos scores had been release that SWEbench pro had major contamination and that Mythos had memorized many questions that lacked the context to be solvable on their own. And now with OpenAI saying a large number of the questions are broken, I think it's worth taking that single outlier benchmark with some salt when the overall trend is that 5.6 is very competitive with Mythos at about half the price.

▲

mchinen 2 hours ago | parent | prev | next [-]

I totally missed that, because in the charts they showcase for coding, the SWEBench score is not present, they only include it at the end of the post in tables. Hmm.

Great catch.

▲

saberience 2 hours ago | parent [-]

The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.

	▲	mnicky an hour ago \| parent [-]
		SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.

▲

simianwords 2 hours ago | parent | prev | next [-]

> SWEBench Pro, which is a pretty high-quality benchmark

No, doesn't seem like it

https://openai.com/index/separating-signal-from-noise-coding...

▲

lwansbrough 2 hours ago | parent | prev [-]

Didn't they also just post about how SWEBench is broken?