OK, but that doesn't address the question of what can be inferred from Moonshot apparently having access to far less compute then the western labs. To what extent does this reflect the need for less compute due to the (published) architectural efficiencies of their model, and to what extent that they just trained for longer (esp. pre-training - 80% of total compute, perhaps)?
I think the word "distillation" needs to be used a bit more selectively here. If their pre-training run was complete before Fable was released that implies that ZERO Fable data went into the base model. Perhaps the timeline allows for a few weeks at best of incremental post-training on some limited amount of Fable data, but calling this "distillation" seems a bit dramatic especially given the redacted outputs that would have been available. A more factual speculation would just be that they may have had time to post-train using a limited amount of Fable output in some fashion (LLM as judge? SFT? Who knows ...).
