Looks like there is a memory hierarchy [1] even in LLMs.
[1] https://en.wikipedia.org/wiki/Memory_hierarchy