Remix.run Logo
xfalcox 4 days ago

Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet.

jgalt212 4 days ago | parent [-]

and you don't have to re-embed everything if the provider sunsets a model.