Crushing it on DeepSWE is a very big deal. Excited to give this a try.

I know everyone is benchmaxxing but this one feels one step too far. Doesn't DeepSWE have both public and private tasks? I'd love to see the diff here.

It looks more like Google execs losing their mind and pressuring researchers to put DeepSWE directly into the training set.

▲

re-thc 8 hours ago | parent | prev [-]

> DeepSWE is a very big deal

It's clearly been "dealt with" already. When it launched we had interesting gaps and definitely differences. Now every new release is "crushing it".

	▲	ttul 6 hours ago \| parent [-]
		Will look forward to the "feel" of the model in real testing. But I agree that these benchmarks do get "dealt with" rapidly. That's a shame, but I guess it's the times we live in.