| |
| ▲ | tristanj 3 days ago | parent [-] | | Moonshot already has at least several million exchanges distilled from Claude that they obtained over the past year https://www.anthropic.com/news/detecting-and-preventing-dist...
. So they have the infrastructure already set up to do this. Re-running their existing distillation suite on the new Fable endpoint would be trivial. Given that Fable was available for 72 hours back in June, I asked GPT-5.6 Sol to estimate how many accounts are needed to generate 1–2 million exchanges with Fable within that timeframe. It concluded it is achievable with only a few hundred accounts. Here's GPT-5.6's conclusion: Under a deliberately simplified, compliant planning model, a Max 20x account could produce approximately 1,944 to 5,832 standard exchanges during 72 hours when Fable 5 use is restricted to 50% of the modeled subscription capacity. The central planning estimate is 3,888 exchanges per account. The 1 to 2 million exchange target is therefore reachable in the central case with roughly 257 to 515 accounts. And the analysis: https://markbin.net/s/pd_9DpYsQr9/sh_vxwgtYdQ?sig=9552292ab3... | | |
| ▲ | amluto 2 days ago | parent | next [-] | | As far as I know, 100% of those Fable interactions would have had encrypted reasoning blocks, so distillation would be distinctly nontrivial even if the data were somehow available. | | |
| ▲ | tristanj 2 days ago | parent [-] | | Encrypted reasoning traces don't prevent distillation. You only need the input prompts and final output responses to distill capabilities. As Anthropic explained in their distillation report (linked above), Moonshot AI already collected millions of session traces, covering: * Agentic reasoning and tool use * Coding and data analysis * Computer-use agent development * Computer vision Hiding the internal CoT blocks stops you from training on internal reasoning traces, sure, but it does nothing to prevent standard input-output distillation. Plus, the CoT blocks weren't even removed/hidden completely. They're still visible, just in summarized form. Raw CoT was replaced by summarized CoT, and summarized CoT still has distillation value. | | |
| ▲ | amluto 2 days ago | parent [-] | | How would you propose to distill Fable-style input/output pairs without the CoT? If you use them as SFT input, you’ll be trying to train a model to predict the post-reasoning output without any reasoning, and this seem very unlikely to work at all with the size of model that Kimi produced and the complexity of Fable’s output. You can’t really “RL” with them because they would be so far off policy that there would be nothing to reinforce. I suppose you could feed input/output pairs to a teacher model and attempt to generate reasoning traces, but it seems like some wishful thinking would be required to get anything even close to as good as Kimi K3 out. Maybe Kimi used these traces to generate RL gym-style problems and somehow produced an evaluator based on the outputs? They would not have had a lot of time in which to do this, and the learning style would not even remotely resemble that which Anthropic used to train Mythos/Fable in the first place. But what do I know? I’m not an expert here. | | |
| ▲ | tristanj 2 days ago | parent [-] | | Anthropic removed raw CoT from all models since Sonnet 3.7 (released February 2025), all Opus 4.X models have censored CoT, yet Chinese labs continue to distill from them, proving this restriction wasn't an insurmountable hurdle. There are some outputs that are highly valuable and cannot be censored, such as tool API calls or agentic tool usage, which is precisely what Moonshot was accused of harvesting/distilling. Tool call harvesting was enough of an issue that Anthropic began inserting spurious tool calls when they detected distillation attempts, trying to poison the distilled data. There is certainly more data the labs are training off of, but that's difficult to know without insider information. |
|
|
| |
| ▲ | boesboes 2 days ago | parent | prev [-] | | What do you base that 1-2 million on? Sounds like complete horseshit imo | | |
|
|
