Are you doing CPU inferencing? For small models you could check qxotic.ai which is a whole inferencing stack for many popular open models in Java, which can beat llama.cpp in speed in some cases. Its tokenizer toknroll is faster than tiktoken, which is what made me think of it for your use case.

The advantage is that you don't have issues like the GIL or whatever, so there's no need for all the complexity of load balancing between multiple pythons.

Thanks for the pointers, I'm gonna look into them.

I'm not doing CPU inferencing but I'm doing some tokenization in Python which is CPU-bound. In all honesty I started on this project to scale the model and did not build it myself so have been trying to read code and walk back the approach. I've been thinking about rewriting the Python glue in Go to avoid issues with the GIL.