Remix.run Logo
thehamkercat 3 hours ago

Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?

Source?

hedgehog 2 hours ago | parent | next [-]

See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).

try-working 2 hours ago | parent | prev [-]

edit: updated the answer above to be more qualitative instead