About the caching things. This is a flaw in the backend, not in the Frontend. If the backend is only relying on prefix cache you need to look for more fine grained cache solutions like HiCache or Lmcache. If your backend discards the whole cache because of one date is changed, you really can’t blame it on the Frontend. It is an implementation detail that you push to the wrong side.

That's not how it works. While there are some research papers looking into using chunks of kv cache while discarding others, those involve architecture changes in the models themselves. HiCache / LMcache (used in sglang/vllm) are ways of optimising the juggling of kv cache, but IIUC it still needs to be token perfect atm.

Prefix caching is inherent in how global softmax attention works. Modifying the prefix invalidates the suffix because the suffix's cached KV projections are a function of the hidden state after the model has evaluated the prefix.

There have been some attempts to approximately stitch KV blocks without full re-evaluation (see: CacheBlend on arxiv), but the results aren't promising.

You mentioned HiCache: as far as I can tell this is a prefix tree cache that efficiently shares prefixes across related histories. If you change the prefix, everything after the change still misses the cache and needs to be prefilled again.