Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.

I could of course download and test myself, but that would take days with my internet connection.

▲

InvertedRhodium 25 minutes ago | parent [-]

I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.

I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.

After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.

Q4_K_P

	▲	Balinares 3 minutes ago \| parent [-]
		I anecdotally observed the same. Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.