fwiw, if I were evaluating these proxies against each other, I would be intrigued by the solution built by people from the Envoy team. Envoy is great software and I’m sure there are many lessons you took from building it.

It looks like you’re even building on Envoy as the foundation for the system which just makes it more compelling.

Its a core dependency for rate limiting, traffic shaping, fail over detection. Its cluster subsystem is super convenient for local LLM calls too. We'll write up a blog on the lessons because there were many. For example, for intelligent routing decisions we can't create an upstream connection to a cluster based on route paths or host - Envoy forces a more static binding. This doesn't work when you are making decisions about a prompt and have to inject more dynamic flow control.