the machines arent optimized for it is why. the main driving factor is large unified memory which makes large(r) models possible, but there isnt the gpu horsepower to back it up. essentially, it fits the corner of the market that wants large models and is ok with running them slowly which doesnt sound like it would be a large market.

Macs have excellent generation speed, and the new Ultra will positively smash that at 1.2TB/sec of bandwidth. For example, that new 176B parameter Qwen model would generate tokens at ~200 tokens/sec.

Macs don’t have very good prefill, though. So it’s important to use a model serving stack that has excellent prompt caching and use a harness that won’t bust the cache.

I’m cross-shopping DGX Sparks and M5 Studios, and having a hard time deciding because they have exactly opposite characteristics for prefill and decode.

For decode, memory bandwidth is the main bottleneck, so these machines will likely perform well even without a ton of GPU horsepower. Not as well as Blackwell, but I expect they will be a reasonable choice in terms of price/performance if you want to run large models with a lot of context.

The main place they are a bit behind is in the number formats they support natively. Iirc M5 doesn't have native FP8 support, so you will take a speed penalty on quants where other architectures get better acceleration.