It's not optimised deliberately for English, right? It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.

▲

numpad0 25 minutes ago | parent | next [-]

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?

▲

Phemist 2 hours ago | parent | prev | next [-]

Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script are widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). So all Serbian text is in a three-way split between these scripts.

Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)

* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

▲

sznio 2 hours ago | parent | prev | next [-]

logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

	▲	brainwad 2 hours ago \| parent [-]
		Most CJK characters are 3 bytes in UTF-8. But then most English words translate to more than one character, so 5 bytes might be more realistic (apparently Chinese averages 1.6-1.7 characters per word); for comparison English gets like 4.7 characters/bytes per word on average.

▲

vova_hn2 2 hours ago | parent | prev | next [-]

> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

▲

cubefox 2 hours ago | parent | prev [-]

They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.