The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood

Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache.

Try to leave only a gigabyte or two free, speed likely would drop dramatically.

Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.