Remix.run Logo
marcosdumay 3 hours ago

The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.