> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

▲

GeekyBear an hour ago | parent | next [-]

The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.

> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.

https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...

	▲	ambicapter 33 minutes ago \| parent [-]
		Extremely impressive.

▲

wongarsu an hour ago | parent | prev | next [-]

My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads

If that's true, inference speed would be even lower if you have only 2GB total, including OS caches

▲

Schiendelman 14 minutes ago | parent | prev | next [-]

Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?

	▲	tracker1 9 minutes ago \| parent [-]
		IIRC, depends on the SSD size.. larger sizes had 2x the bandwidth, so it depends. That's combined/offset with the M5 improvements even further.

▲

afzalive an hour ago | parent | prev | next [-]

The M5 MBP has 24GB of RAM, more context in RAM perhaps?

▲

gitpusher42 an hour ago | parent [-]

The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood

	▲	petu 24 minutes ago \| parent [-]
		Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache. Try to leave only a gigabyte or two free, speed likely would drop dramatically. Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.

▲

gitpusher42 an hour ago | parent | prev [-]

[dead]