Isn't part of this the inevitable cache invalidation that comes from switching between providers (Opus to Gemini)?

Yeah, but you only pay the switch on the much cheaper model, the first bit is done with smaller context on the expensive model, so you still save money in the end.