Thanks for sharing. Makes a lot of sense that removing that routing layer would improve e2e latency.

We had a similar bottleneck building out our sandbox routing layer, where we were doing a lookup to a centralized db to route the query. We found that even with a fast KV store, that lookup still added too much overhead. We moved to encoding the routing logic (like region, cluster ID, etc) directly into the subdomain/hostname. This allowed to drop the db read entirely on the hot path and rely on Anycast + latency-based DNS to route the user to the exact right regional gateway instantly. Also, if you ever find yourselves outgrowing standard HTTP proxies for those long-lived agent sessions, I highly recommend looking at Pingora. It gave us way more control over connection lifecycles than NGINX.

For the compute aspect doing sandbox pooling is cool but might kill your unit economics, especially if at some point each tenant has different images. Have you looked into memory snapshots (that way you only have storage costs not full VMs)?