The author put in the numbers, but maybe you didn’t read them.

45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio.

Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash

▲

mike_hearn 2 minutes ago | parent | next [-]

I think it's been pretty much proven by now that there are no cases where local inferencing is better than remote inferencing, unless absolute privacy is a hard requirement. The efficiencies that come with datacenter scale and hw can't be beaten.

▲

sho 4 hours ago | parent | prev [-]

The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous.

There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.

▲

EagnaIonat an hour ago | parent [-]

> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.

Your knowledge is out of date. In truth it depends on the Mac and the models used.

I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.

Question: "Give me a python code snippet that opens a file and sorts the lines of text. "

In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.

Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.

	▲	desterothx an hour ago \| parent [-]
		We are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter