Remix.run Logo
▲ mugul 2 hours ago

I think it is: look at the "in" which is likely one token, "strawberry" looks like it is 2x the length of "in". Same for the last word, "ing" seems less compressed than "bear" right before it.

▲dahart 2 hours ago | parent [-]

Looking at the first several examples, shorter (one, two, three-) letter words are not stretched to the full ‘fixed’ width. It’s the whitespace around them that is fixed. The only example in the first three screenshots of splitting a word into tokes is ‘vibes’ which turns into ‘vib’ and ‘es’. I’m skeptical of the -ing and -tion words being single tokens as shown.