Did opus have an oracle for the original compiler? If not, then it's expected that qwen would do a better job.

As is known, LLMs are amazing when verification is cheap, and much less useful when it's expensive.

> Did opus have an oracle for the original compiler?

It actually did. There was an older yacc-based implementation that had a lot of architectural limitations. We decided that it was better to start over with a better architecture, but we did have that older one around to compare generated microcode initially.

	▲	disgruntledphd2 2 hours ago \| parent [-]
		Ok, that's interesting. My prior would have been that if Opus didn't have an oracle, then there's still a pretty large gap. But if, as you say, both had an oracle then this is a really interesting outcome. To be fair, I think the differences between frontier and open models is pretty small (I've been swapping back between Claude and GLM over the last week). The one thing that I notice is that GLM keeps trying to blame external factors for any non-matches (we're rebuilding an external tool for internal use), whereas Claude adheres to my prompt much better (i tell it to only blame these factors if it's ruled out everything else). I'm not convinced that change is worth 3.5-5x cost differentials though (I only used Claude because we have the small business package, which is way cheaper than API rates).