I totally missed that, because in the charts they showcase for coding, the SWEBench score is not present, they only include it at the end of the post in tables. Hmm.

Great catch.

▲

saberience 2 hours ago | parent [-]

The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.

	▲	mnicky an hour ago \| parent [-]
		SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.