| ▲ | paulryanrogers an hour ago | |
If tokens use UTF-8 bytes then it could be a factor. | ||
| ▲ | ben_w an hour ago | parent [-] | |
A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'. | ||
| ▲ | paulryanrogers an hour ago | |
If tokens use UTF-8 bytes then it could be a factor. | ||
| ▲ | ben_w an hour ago | parent [-] | |
A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'. | ||