| ▲ | colingauvin 6 hours ago |
| Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence. |
|
| ▲ | voxic11 6 hours ago | parent | next [-] |
| Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531 Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit |
| |
| ▲ | jmalicki 4 hours ago | parent | next [-] | | That distillation exists isn't the question. It's about evidence this is an active force in competition in LLMs. | | |
| ▲ | mindwok 3 hours ago | parent [-] | | I think the biggest actual piece of evidence is how hard the major players are trying to stop it |
| |
| ▲ | maleldil 5 hours ago | parent | prev | next [-] | | I took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven. | | |
| ▲ | sebzim4500 5 hours ago | parent [-] | | I doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models. | | |
| ▲ | colingauvin 5 hours ago | parent | next [-] | | I'm saying that it is unclear that without distillation this wouldn't still be happening. There is a massive narrative that no one but OpenAI, Anthropic, and Google can make a model without distilling. But there's basically no evidence of that. | |
| ▲ | miki123211 5 hours ago | parent | prev [-] | | Alternatively, modern AI is good enough at optimizing its own kernels that it just keeps pushing costs down. Unlike the semi-decentralized inference provider community, OpenAI has both the talent and the compute to throw at the problem of making their models much more efficient to run. GPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves. |
|
| |
| ▲ | porridgeraisin 2 hours ago | parent | prev [-] | | To be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO. |
|
|
| ▲ | hadlock 2 hours ago | parent | prev | next [-] |
| It turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it. |
|
| ▲ | endymi0n an hour ago | parent | prev | next [-] |
| Been using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release. |
| |
|
| ▲ | flexagoon 3 hours ago | parent | prev | next [-] |
| Why does Kimi insist its name is Claude? |
| |
| ▲ | Paradigma11 2 hours ago | parent [-] | | Please elaborate the mechanisms by which a LLM would know what model it is. | | |
| ▲ | Sabinus an hour ago | parent | next [-] | | By being trained on text containing "I am X" in the model response section. | |
| ▲ | an hour ago | parent | prev | next [-] | | [deleted] | |
| ▲ | datadrivenangel an hour ago | parent | prev [-] | | ask it what type of model it is or what it's name is... it's weird that Kimi will say it's claude... | | |
| ▲ | InvertedRhodium an hour ago | parent | next [-] | | The training data likely references Claude significantly more often than Kimi, given the popularity of the models. There will simply be more examples of “Claude” being the response to that question. | |
| ▲ | NekkoDroid 32 minutes ago | parent | prev | next [-] | | Doesn't Claude say its Deepseek when asked in Chinese? I remember there being posts about that a while ago. | |
| ▲ | tshaddox an hour ago | parent | prev [-] | | Okay, but I would also ask “why does Claude say that its name is Claude?” |
|
|
|
|
| ▲ | v64 2 hours ago | parent | prev | next [-] |
| The evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting. [1] https://www.anthropic.com/news/detecting-and-preventing-dist... |
|
| ▲ | InsideOutSanta 6 hours ago | parent | prev | next [-] |
| Musk confirmed in federal court that xAI does it:
https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x... It's also how providers build their smaller models out of their larger ones; they publicly talk about the process. |
|
| ▲ | NicoJuicy 6 hours ago | parent | prev | next [-] |
| Here's an example: https://github.com/microsoft/Build25-LAB329 |
|
| ▲ | JacobAsmuth 6 hours ago | parent | prev | next [-] |
| [flagged] |
|
| ▲ | retinaros 6 hours ago | parent | prev [-] |
| there is no evidence. it shortcuts post training by a huge margin this is true. but that is all. |
| |
