▲ | astrange 8 days ago | |
Tokenization makes things harder, but it doesn't make them impossible. Just takes a bit more memorization. Other writing systems come with "tokenization" built in making it still a live issue. Think of answering: 1. How many n's are in 日本? 2. How many ん's are in 日本? (Answers are 2 and 1.) |