I think that switching from fixed token sets to something like H-Nets [0] (discussed on HN [1]) can at least improve on this issue, if not solve it completely.

Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.

[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1

[1] https://news.ycombinator.com/item?id=50278253

▲

cubefox 3 hours ago | parent [-]

Exactly. Currently LLMs use a fixed tokenizer optimized for English and (to a lesser degree) Western languages. The H-Net architecture doesn't use a fixed tokenizer, it uses dynamicaly adaptive chunks of bytes, and (in a two-stage setup) chunks of chunks of bytes.

However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.

▲

brainwad 2 hours ago | parent | next [-]

It's not optimised deliberately for English, right? It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.

▲

numpad0 29 minutes ago | parent | next [-]

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?

▲

Phemist 2 hours ago | parent | prev | next [-]

Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script are widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). So all Serbian text is in a three-way split between these scripts.

Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)

* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

▲

sznio 2 hours ago | parent | prev | next [-]

logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

	▲	brainwad 2 hours ago \| parent [-]
		Most CJK characters are 3 bytes in UTF-8. But then most English words translate to more than one character, so 5 bytes might be more realistic (apparently Chinese averages 1.6-1.7 characters per word); for comparison English gets like 4.7 characters/bytes per word on average.

▲

vova_hn2 2 hours ago | parent | prev | next [-]

> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

▲

cubefox 2 hours ago | parent | prev [-]

They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.

▲

stephantul 2 hours ago | parent | prev [-]

Non-latin languages also take more bytes due to utf-8 though

▲

ben_w 2 hours ago | parent | next [-]

Are bytes relevant? These things are billed by the megatoken, not the megabyte.

▲

paulryanrogers an hour ago | parent [-]

If tokens use UTF-8 bytes then it could be a factor.

	▲	ben_w an hour ago \| parent [-]
		A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.

▲

numpad0 an hour ago | parent | prev | next [-]

bit configurations aren't uniform across those bytes, everything should compress to similar sizes

▲

guipsp 2 hours ago | parent | prev [-]

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm