| ▲ | FrancescoMassa a day ago | |||||||
You transfer context from the kv cache of model A to the kv cache of model B | ||||||||
| ▲ | verdverm a day ago | parent [-] | |||||||
you cannot do this with direct byte copy, you have to reprocess all input to rebuild the cache in model B, it's exactly like waiting hours to send a new message to an existing session and your cache has expired tl;dr - you will not get the 90% discount the first time you change models and send a message this is why I am saying you cannot "transfer" kv cache, it's just like changing models in your favorite harness and that requires reprocessing the input from scratch to build a kv cache for the model | ||||||||
| ||||||||