Thanks for the pointers, I'm gonna look into them.

I'm not doing CPU inferencing but I'm doing some tokenization in Python which is CPU-bound. In all honesty I started on this project to scale the model and did not build it myself so have been trying to read code and walk back the approach. I've been thinking about rewriting the Python glue in Go to avoid issues with the GIL.