Ok, that's interesting. My prior would have been that if Opus didn't have an oracle, then there's still a pretty large gap.

But if, as you say, both had an oracle then this is a really interesting outcome.

To be fair, I think the differences between frontier and open models is pretty small (I've been swapping back between Claude and GLM over the last week).

The one thing that I notice is that GLM keeps trying to blame external factors for any non-matches (we're rebuilding an external tool for internal use), whereas Claude adheres to my prompt much better (i tell it to only blame these factors if it's ruled out everything else).

I'm not convinced that change is worth 3.5-5x cost differentials though (I only used Claude because we have the small business package, which is way cheaper than API rates).