I don't like that it throws away the whole KV cache when compacting. It costs a cache miss of the whole conversation length, and that's a waste of time and money.

LLMs are perfectly capable of summarising the conversation without a new system prompt.

Yeah, that seems suboptimal, unless you're using a much cheaper model to do compaction.

How the summarizing of the conversation happens in an LLM?