▲ | xfalcox 4 days ago | |
Having a public tokenizer is quite useful, specially for embeddings. It allows you to do the chunking locally without going to the internet. | ||
▲ | jgalt212 4 days ago | parent [-] | |
and you don't have to re-embed everything if the provider sunsets a model. |