Remix.run Logo
danudey 17 hours ago

Sort of? It's about locality of data; this has often been a bottleneck, which is why we have CPU caches to keep data extremely close to the CPU cores with practically zero latency and throughput limitations compared to fetching from main memory. Unfortunately now we're shuffling terabytes of data through our algorithms and the CPU spends a huge amount of its time waiting for the next batch of data to come in through the pipe.

This is, IIRC, part of why Apple's M-series chips are as performant as they are: they not only have a unified memory architecture which eliminates the need to copy data from CPU main memory to GPU or NPU main memory to operate on it (and then copy the result back) but the RAM being on the package means that it's slightly "more local" and the memory channels can be optimized for the system they're going to be connected to.