| ▲ | milkshakes 5 hours ago |
| assume you are a "second class lab" and you are in fact making progress by distilling the results of the frontier labs' efforts. what is the end game for this strategy? if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress? |
|
| ▲ | InsideOutSanta 5 hours ago | parent | next [-] |
| This line of thinking makes no sense because it assumes that labs that distill from frontier models are doing nothing else. It's the classic "the Chinese can only copy" mentality, and it's going to end poorly for American companies. I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications. |
| |
| ▲ | milkshakes 4 hours ago | parent [-] | | i never assumed that, and i do keep up with the publications. i'm also not saying it's a dumb thing to do! what i am saying is that empirically, it appears that distillation of a more advanced model is a required first step for them to train a borderline competitive, cheaper model. in effect, their training is subsidized by the frontier labs. if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off? | | |
| ▲ | InsideOutSanta 4 hours ago | parent [-] | | > empirically, it appears that distillation of a more advanced model is a required first step I see no evidence for that. > if this were not the case, then we would be observing chinese models that far surpass frontier models It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI. > what happens to these efforts when the subsidy is cut off? They will continue making progress as they do now, minus the benefits of distillation. | | |
| ▲ | milkshakes 4 hours ago | parent [-] | | https://www.anthropic.com/news/detecting-and-preventing-dist... Moonshot AI
Scale: Over 3.4 million exchanges The operation targeted: Agentic reasoning and tool use
Coding and data analysis
Computer-use agent development
Computer vision
Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces. | | |
| ▲ | InsideOutSanta 3 hours ago | parent | next [-] | | I'm assuming you posted that as evidence for the claim that "empirically, it appears that distillation of a more advanced model is a required first step", but I don't think it is. It's just evidence that Moonshot distills Anthropic's models, which, yes, they do. | | |
| ▲ | milkshakes 3 hours ago | parent [-] | | it is not a required first step for training a model, sure. but that's not what i claimed. what i claimed is that is how they are so significantly _reducing the cost_ of training one! how else do you think they are doing it? |
| |
| ▲ | idiotsecant 3 hours ago | parent | prev [-] | | >request metadata, which matched the public profiles of senior Moonshot staff Translation: we have the machinery in place to identify our users, and actively do so. |
|
|
|
|
|
| ▲ | tristanj 2 hours ago | parent | prev | next [-] |
| Distillation from a teacher model solves the self-start problem, that is, building a model to the point where it reason coherently. Without distillation, solving self-start is incredibly difficult since it requires millions of high quality training samples. Creating that kind of dataset takes an enormous amount of effort. Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI). Kimi K3 is capable of RSI. |
|
| ▲ | traverseda 5 hours ago | parent | prev | next [-] |
| In public with budgets that don't risk destroying the American economy presumably. Yes it may be slower. |
| |
| ▲ | milkshakes 5 hours ago | parent [-] | | > with budgets and what will fund these budgets exactly?
inference is cheap, distillation is cheap, training is what's expensive. | | |
| ▲ | traverseda 5 hours ago | parent | next [-] | | Same people who fund linux kernel development. A coalition of companies that find it useful. | |
| ▲ | Jtarii 5 hours ago | parent | prev [-] | | Presumably the US military / NSA. | | |
| ▲ | milkshakes 5 hours ago | parent [-] | | the USG/NSA will fund chinese labs?
to what end? | | |
| ▲ | Jtarii 5 hours ago | parent [-] | | I was more thinking they would be funding US labs. | | |
| ▲ | milkshakes 5 hours ago | parent [-] | | the question was: what is the endgame for the stated "second class labs" strategy of distilling their frontier competitors then undercutting them on price? | | |
| ▲ | traverseda 4 hours ago | parent | next [-] | | Yes yes, we all understand the game-theoretic race-to-the-bottom you're describing here. Somehow despite linux being FOSS it still powers most of the important computing in the world. Can you explain how that works despite it being free? Once you understand that case I think you'll understand the game-theory behind how large projects can exist in the absence of traditional IP protection. | | |
| ▲ | milkshakes 3 hours ago | parent [-] | | the obvious difference is the massive scale of data and compute required to develop and evolve these models, and the costs they impose on those building them. | | |
| ▲ | traverseda an hour ago | parent [-] | | Smaller budgets, slower improvement, less risk. They're not entitled to profits if that business model isn't sustainable. They're not entitled to a change in IP laws to protect their business model. They're not entitled to growing that fast. |
|
| |
| ▲ | nickysielicki 4 hours ago | parent | prev [-] | | Making lots of money? |
|
|
|
|
|
|
|
| ▲ | IncreasePosts 5 hours ago | parent | prev [-] |
| There doesn't need to be progress at this point. Some models even from 1 or more years ago are useful for some purposes |
