Memorilla: Latent Semantic Memory for LLMs

▲ Memorilla: Latent Semantic Memory for LLMs(arxiv.org)

74 points by handfuloflight 5 hours ago | 5 comments

▲ jkingsman 5 hours ago | parent | next [-]

So if I'm reading this right, it's basically a specialized embedding of a RAG, and then a model in the middle takes the agent's queries and returns responses in a form that is injecting into the main model's input token stream directly i.e. skipping actual text. And the injection format is INCREDIBLY token efficient. Sort of injecting a vibe rather than quotable text. Their results of "good on concepts, less good on quotes/exact text retrieval" seem to intuitively line up with that.

▲ NaiveEmpiricist 3 hours ago | parent | next [-]

It looks like another difference with RAGs is that instead of reranking the doc vectors and taking the top 5 or so, this paper mixes all the docs (weighted by relevance) into a fixed number of "memory tokens".

It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.

	▲	CharlieDigital an hour ago \| parent [-]
		`> ...a problem I've had with RAG - when a fact isn't local to a single patch of text` Is this a problem, though? Vector similarity should surface the relevant, related patches.

▲ johnshahbazian 3 hours ago | parent | prev [-]

I think its offering a compromise between accuracy and tokens. You don't have to use as many tokens and you get pretty good accuracy, but not the absolute best. I think, however, in many situations people aren't super concerned about the tokens as they are with the accuracy.

▲ bob1029 an hour ago | parent | prev [-]

I am finding that semantic ~= lexical when you have something like a frontier model driving. Using full text search engines is still the most compelling approach for handling large document sets.

I really struggle with the semantic / embedding concerns when we have a generative model that can reverse engineer a Mario 64 rom dump to run on my toaster oven. These models know how to compose Lucene & other FTS query strings in such a way that we are effectively getting semantic search over purely lexical systems.