As far as I know, 100% of those Fable interactions would have had encrypted reasoning blocks, so distillation would be distinctly nontrivial even if the data were somehow available.

▲

tristanj 2 days ago | parent [-]

Encrypted reasoning traces don't prevent distillation. You only need the input prompts and final output responses to distill capabilities. As Anthropic explained in their distillation report (linked above), Moonshot AI already collected millions of session traces, covering:

* Agentic reasoning and tool use

* Coding and data analysis

* Computer-use agent development

* Computer vision

Hiding the internal CoT blocks stops you from training on internal reasoning traces, sure, but it does nothing to prevent standard input-output distillation.

Plus, the CoT blocks weren't even removed/hidden completely. They're still visible, just in summarized form. Raw CoT was replaced by summarized CoT, and summarized CoT still has distillation value.

▲

amluto 2 days ago | parent [-]

How would you propose to distill Fable-style input/output pairs without the CoT?

If you use them as SFT input, you’ll be trying to train a model to predict the post-reasoning output without any reasoning, and this seem very unlikely to work at all with the size of model that Kimi produced and the complexity of Fable’s output. You can’t really “RL” with them because they would be so far off policy that there would be nothing to reinforce. I suppose you could feed input/output pairs to a teacher model and attempt to generate reasoning traces, but it seems like some wishful thinking would be required to get anything even close to as good as Kimi K3 out.

Maybe Kimi used these traces to generate RL gym-style problems and somehow produced an evaluator based on the outputs? They would not have had a lot of time in which to do this, and the learning style would not even remotely resemble that which Anthropic used to train Mythos/Fable in the first place.

But what do I know? I’m not an expert here.

	▲	tristanj 2 days ago \| parent [-]
		Anthropic removed raw CoT from all models since Sonnet 3.7 (released February 2025), all Opus 4.X models have censored CoT, yet Chinese labs continue to distill from them, proving this restriction wasn't an insurmountable hurdle. There are some outputs that are highly valuable and cannot be censored, such as tool API calls or agentic tool usage, which is precisely what Moonshot was accused of harvesting/distilling. Tool call harvesting was enough of an issue that Anthropic began inserting spurious tool calls when they detected distillation attempts, trying to poison the distilled data. There is certainly more data the labs are training off of, but that's difficult to know without insider information.