I spent three weeks debugging why my Qwen 122B setup on an M3 Ultra was taking 3–5 minutes to generate the first token on follow-up messages (despite having a "warm" context).

The root cause wasn't the model, but three specific infrastructure bugs in my serving stack:

1. Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing a full re-compute every turn.

2. Interrupt Path: Streaming replies weren't persisted when the generation was interrupted, causing history divergence.

3. Checkpoint Poison: A background writer created unmatchable checkpoints that crowded out valid ones, triggering aggressive eviction.

After fixing these, prefill time dropped from minutes to sub-seconds (53k tokens cached, 33 tokens prefilled).

I've open-sourced the fork (qMLX) and a benchmarking tool to verify these numbers. Would love feedback on the hybrid attention caching strategy or any other edge cases I might have missed.