We just did exactly this - added TypeSafe-compatible API (incl. websocket support) for various VLMs. Latency right now is <250ms, but will be able to get it to <150ms (p95).

Take a look at a snake demo with streaming image inputs: https://x.com/spillai/status/2103630735425089957