This is why I keep saying Chinese models are nowhere as good as their benchmark scores suggest, and MaxxBench seems to be very compelling evidence of that.

Every time a Chinese lab releases a new model, I give it a solid try that same weekend and drive it hard by giving it the exact same tasks that I give to Claude/Codex models. And the new releases all have the same issues: they start out strong, but start deviating after about ~30 minutes and I quickly find myself in a quagmire where the model makes mistake after mistake while fixing its previous mistakes. Sometimes these are minor. Occasionally they can be pretty bad. Like, would-cause-data-loss-if-merged bad.

Ultimately I also reject the notion that Chinese labs are only a few months behind US labs. We haven't had an actual Fable-level model come out of China. AFAIK only Kimi 3 was claimed to come close, and then it turned out that not only had they heavily distilled Anthropic models, but, unbeknownst to their users, they were routing some percentage of their requests to Fable.

Wouldn't benchmark exactly show that the open models are only a few months behind the Anthropic ones? GLM 5.3 sits a few points behind Opus 5, and these models came out within about a month. Opus 5 was July 24th and GLM 5.3 was August 14th.

Obviously these models are benchmaxxed, and it's frustrating to see not only so much attention on these benchmarks, from both users and the labs, but there is a difference between "The models are benchmaxxed" and "that means the open models are much worse than the closed ones"

A lot of users here seem to draw a correlation between these two, but what the article shows mostly is that the smaller models are more benchmaxxed than the bigger ones. That includes both open weights and closed weight models, as Luna seems to be among the worst benchmaxxed models. It's just that the open weights models are smaller than the closed weights ones.

Obviously though, every benchmark has to be taken with a grain of salt, including this one. But we have to be careful to not cherry pick the conclusions that prove our pre-existing notions