My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads

If that's true, inference speed would be even lower if you have only 2GB total, including OS caches