| ▲ | dahart 2 hours ago | |||||||
Cute idea, but is this fixed width words, and not fixed width tokens? Isn’t an ‘ing’ ending normal a separate token from the root? Isn’t ‘strawberry’ two tokens? | ||||||||
| ▲ | mugul 2 hours ago | parent [-] | |||||||
I think it is: look at the "in" which is likely one token, "strawberry" looks like it is 2x the length of "in". Same for the last word, "ing" seems less compressed than "bear" right before it. | ||||||||
| ||||||||