Remix.run Logo
lnenad 4 hours ago

Considering the DeepSWE result (imho if you're gonna give value to benchmarks this is one of the best) it's not good enough.

svantana 3 hours ago | parent [-]

It's a high quality benchmark for sure, but it being public means it's at risk of leaking into the models (unintentionally or not), right? For that reason I prefer to look at the private ones, like: HLE, SimpleBench, Kagi, ARC-AGI.