Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.

SWE-Bench Pro was created to replace SWE-Bench and fix these problems.

SWE-bench Verified was created to fix the problems of SWE-bench.

Then SWE-Bench Pro was created because SWE-bench Verified had flaws.

Now SWE-Bench Pro is shown to have flaws.

carabiner 7 hours ago | parent | next [-]

Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?

	▲	jaggederest 6 hours ago \| parent [-]
		Bench Bench Pro Maxx Series S 360? The original Bench Bench Pro Maxx Series S had some quality issues, so that's the current followup. We've also released a higher order benchmark developed out of Bench Bench Pro Maxx Series S 360 One King Ranch edition, allowing future benchmark towers to be fully self-contained.

Well, we now have DeepSWE