My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.

It's also mapped into the address space so there's very little extra latency in grabbing the embedding as opposed to something like nvme that will have to setup a command list, submit it to the drive's microcontroller, wait for the op to be processed, etc.