| ▲ | cubefox 2 hours ago | |
They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283 I don't know whether European languages work inherently better with tokenizers, but I don't see why they would. | ||