My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`

With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work

And common weights still use mmap for simplicity

So, I believe llama.cpp might run it under 2gb, but I assume it will be slower

Any idea if madvise helps? Admittedly I have very limited experience and only on Linux

for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?