Remix.run Logo
▲ origini an hour ago

I would counter argue with "But chinese models use distillation of Frontiermodels..." which is different than being a Frontierlab or not clear how different it is.

But Microsoft coming from the side like this might be very different if it holds up.

▲adrian_b an hour ago | parent | next [-]

For now I have not seen any credible evidence for "But chinese models use distillation of Frontiermodels...".

Of course, I would not be surprised if they also do this, mainly because after OpenAI and Anthropic have succeeded to gather most training information that was easily accessible in the past and now it has become much more difficult to do the same. For some information that was hoarded by OpenAI and Anthropic, public access may have become impossible otherwise than through their LLMs.

Nonetheless, distillation cannot be the main source of improvement for the Chinese models, among other reasons because even assuming that they may have access to a great number of fake accounts, there still is not enough time for a lot of distillation, taking into consideration the timeline of the releases, and it is not believable that the attempts to do distillation would not be detected quickly by OpenAI and Anthropic, leading to the banning of the fake accounts. While OpenAI and Anthropic provide negligible information about their models, the Chinese AI companies are much more transparent about their methods and we can see that there is a lot of diversity among their models and a lot of innovations with each new version.

▲rbehrends 43 minutes ago | parent | next [-]

Yeah, there is a limit to how much you can get from capturing reasoning traces. (Traditional distillation of weights is not possible via the API.)

Patrick Toulme noted a while ago that the biggest advantage of that in his opinion came from bootstrapping the post-training process: https://xxcancel.com/PatrickToulme/status/206921157543762774...

At Thinking Machines, they explicitly said that they used open-weights models for bootstrapping post-training, including Kimi K2.5: https://thinkingmachines.ai/news/introducing-inkling/

Speaking of Thinking Machines, that's another data point. Their Inkling Small 2 model is just a bit behind DeepSeek V4.1 Flash 1017 in benchmark, and in my tests, that seems to be accurate, but mostly because they used less training data for pre-training, so they have less of a knowledge base, while similar reasoning capabilities. (One of the most interesting parts of Inkling Small 2 is that their training must have been very cost-efficient, apparently using less, but higher quality training data.)

And we're now seeing more and more Western near-frontier models from labs other than Anthropic and OpenAI to question whether distilling reasoning traces is really much of a difference maker.

Personally, my theory is that the gap is primarily due to Chinese labs having a harder time getting the necessary compute (especially top of the line GPUs) for training. These are, as far as I can tell, orders of magnitude differences in availability, so it wouldn't be surprising that a large compute gap is reflected in capabilities (and also indirectly visible through model sizes).

▲origini an hour ago | parent | prev | next [-]

I don't think there is perfect proof, but its taken as truth.

Indication for this are for example that Chinese models think they are another model: "chinese model think its claude".

My assumption is, that Chinese model found things Frontierlabs already had but didn't share. There was this paper were i had the feeling that this was a real and significant Research Result from Deepseek "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression" but otherwise its just unclear.

The main issue is, that as long as it seems plausable that they do distillation, its hard to determine how big the gap is.

Not saying they can't catch up but they are also potentially limited by resources (money & nvidia GPUs)

▲rbehrends 34 minutes ago | parent [-]

> Indication for this are for example that Chinese models think they are another model: "chinese model think its claude".

There is evidence that some Chinese labs have been training on frontier model CoT (in particular from the analysis of reasoning traces [1]), but what models claim about their identity is pretty much worthless as evidence. LLMs are not self-aware. These claims come from whatever is in their training data, and Claude/GPT output including such statements is all over the public web, including in some huggingface data dumps. Consider that people got Opus claiming to be Qwen or DeepSeek when asked in Chinese, and you'll see how weak this particular evidence is.

[1] E.g. https://arxiv.org/abs/2608.09867

▲nzd an hour ago | parent | prev [-]

Heck, they destroyed some rare physical books after scanning them. Something to show their mentality.

▲eli 41 minutes ago | parent | prev [-]

Even if OpenAI/Anthropic remain "a few months" ahead of open weight models indefinitely that still is not great news for them.

▲origini 14 minutes ago | parent | next [-]

I currently do not bother with chinese models. I try them out, i use the small ones for small experiments but the price point difference matters so little that I want 'the current best'.

And 6 month feels still very different.

Its for sure an option or might have some kind of impact but if you have some kind of restrictions on what you should use and the price is not that big but the quality and possibilites are better, I can see companies just staying with Anthropic or OpenAI.

You also don't just switch out a model, they might need prompt updates etc.

But we will see I don't mind competition

▲giancarlostoro 19 minutes ago | parent | prev [-]

All it would take is Nvidia finally releasing the 6090's instead of trying to milk enterprise GPUs. I don't understand the blind love for Jensen, he's clearly been screwing over all the consumers who made him and Nvidia wealthy.

Someone somewhere really needs to build a RAM startup that not only makes RAM but disrupts the stronghold AI has over it, it's a little absurd.