Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.

I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.

How do caches work across models? I would have thought that was very model specific - if not I’ve really misunderstood what’s getting cached.

Huh?

Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.