> which multi-model switching doesn't increase

Given model A with cache C(a) and model B with C(b)

Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?

If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.

I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:

* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?

Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.