| ▲ | nullsanity 5 hours ago | |
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that. | ||
| ▲ | marcosdumay 2 hours ago | parent | next [-] | |
The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size. | ||
| ▲ | jascha_eng 3 hours ago | parent | prev | next [-] | |
I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive. | ||
| ▲ | otikik 5 hours ago | parent | prev [-] | |
It’s better for them | ||