| ▲ | thehamkercat 3 hours ago | |
Can you explain how does it work? like how is the previous K/V cache used when you switch to another model? Source? | ||
| ▲ | hedgehog 2 hours ago | parent | next [-] | |
See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model). | ||
| ▲ | try-working 2 hours ago | parent | prev [-] | |
edit: updated the answer above to be more qualitative instead | ||