If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.

I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:

* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?

Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.

[dead]