What is considered SOTA for SWE benchmarks now?

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.

[0] https://deepswe.datacurve.ai/

[1] https://cognition.com/blog/frontier-code-1.1

▲

EuanReid 7 hours ago | parent | prev | next [-]

I've generally found DeepSWE[0] to be pretty true to reality.

[0]: https://deepswe.datacurve.ai/

	▲	7 hours ago \| parent [-]
		[deleted]

▲

swyx 7 hours ago | parent | prev | next [-]

https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)

	▲	sk4rekr0w 4 hours ago \| parent [-]
		1.1 seems a lot better than the original release, which was a bit hyperbolic. excited to see the team keep iterating.

▲

carabiner 7 hours ago | parent | prev | next [-]

strawberry

	▲	retr0rocket 7 hours ago \| parent [-]
		Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry. Unless you want to tack on bpe enconding table to every llm context its pointless

▲

enraged_camel 7 hours ago | parent | prev [-]

FrontierBench

	▲	dandaka 7 hours ago \| parent [-]
		do they have a website? I have found only paper PDF and it seems more general than SWE