Indeed, you might have hundreds of GBs, or even TBs of memory, but caches are still (relatively) small. L1 is going to be in the order of 64k of memory!