... but you'll be rewriting inference for any model that isn't a well-known LLM. Yourself.
AI coding agents can do that pretty nicely already and it will only (slowly) improve over time.