Non-latin languages also take more bytes due to utf-8 though

ben_w 2 hours ago | parent | next [-]

Are bytes relevant? These things are billed by the megatoken, not the megabyte.

If tokens use UTF-8 bytes then it could be a factor.

	▲	ben_w an hour ago \| parent [-]
		A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.

numpad0 43 minutes ago | parent | prev | next [-]

bit configurations aren't uniform across those bytes, everything should compress to similar sizes

guipsp 2 hours ago | parent | prev [-]

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm