Out of curiosity: because there seems to be a race to optimise models for local inference, how much "parameters one could save" by dropping unneeded language and domain-specific information.

Like, can you have a model that is English-only, but does more with the same amount of parameters if Chinese and European languages are dropped from the training?

This is a key question we faced when building this model. It comes down to basically to "how good" to you need to be at "how many things". We had to make some choices with this model and do our best to maximize performance in those areas.

To answer this more precisely its a matter of choosing different data and training regimes and checking performance with evals.

And to make this fully concrete you're welcome to give it a try! Train this model on a taskset of your choice and measure the performance tradeoffs. You'll get a good sense of how LLM capabilities shift

Unfortunately, it doesn't quite work like that. Google this: transfer learning.