Remix.run Logo
burmanm 5 hours ago

DeepSWE is slightly flawed in the sense that is uses only its own harness and that causes issues on models that are not correctly supported by it. There's huge amount of evidence that the harness plays a big role in how these models work and yet DeepSWE entirely removes that (and has probably only tested that it works fine with some favourite model of them).

There's also issues with cost calculation (as that harness doesn't use caches) and so on as reported on their github issues.

None of the benchmarks are perfect, but that does explain a lot of the variations between benchmarks.