Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yet

Quoting RGFusion from Reddit: LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...

Ngram is compressing several layers of multiplication to a lookup which negates the need to have the same model depth and reduces the model size that must be loaded.