I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.

This aligns with the "zero tokens" approach described in this paper. :)

I tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.