| ▲ | ai_slop_hater 5 hours ago | |
Does anyone know what changed in the tokenizer? Does it output multiple tokens for things that were previously one token? | ||
| ▲ | quux 4 hours ago | parent [-] | |
It must, if it now outputs more tokens than 4.6's tokenizer for the same input. I think the announcement and model cards provide a little more detail as to what exactly is different | ||