but nobody is using LLMs all by themselves.

Long-term memory is stored outside the model. In fact, Andrej Karpathy recently talked about the idea that it would be great if we could get LLMs to not know any facts, and that humans poor memory might be a feature which helps with generalization rather than a bug.

This is an interesting idea. I wonder if it's more that we have different "levels" of memory instead of generally "poor" memory though.

I'm reminded of an article on the front page recently about the use of bloom filters for search. Would something like a bloom filter per-topic make it easier to link seemingly unrelated ideas?