Cache is not huge. If you want good performance you need to keep your working set in cache. Sometimes that requires storing offsets (say 32-bit) rather than pointers. If you can also pack info into spare bits, all the better. Perhaps not everyone needs to care about this, but there are still reasons to care.

Indeed, you might have hundreds of GBs, or even TBs of memory, but caches are still (relatively) small. L1 is going to be in the order of 64k of memory!