Remix.run Logo
▲ cubefox 3 hours ago

Exactly. Currently LLMs use a fixed tokenizer optimized for English and (to a lesser degree) Western languages. The H-Net architecture doesn't use a fixed tokenizer, it uses dynamicaly adaptive chunks of bytes, and (in a two-stage setup) chunks of chunks of bytes.

However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.