> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779