Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).

I pretty much just start a new session whenever i fill the context.

In my opinion this is one of the areas where GPUs provide a qualitatively different experience than unified memory boxes.

For an EPYC with a 5090 (no layers on CPU) vs an M3 max 128GB, qwen 3.6 27B at 128k context / 7k generation:

                Cold: prefill + decode    Hot (KV cached)
  5090          40s  + 2-3m  = 3-4 min    2-3 min
  M3 Max 128GB  14m  + 8-10m = 22-25 min  8-10 min