A bit strange to use time to first token instead of throughput.

Latency to the first token is not like a web page where first paint already has useful things to show. The first token is "The ", and you'll be very happy it's there in 50ms instead of 200ms... but then what you really want to know is how quickly you'll get the rest of the sentence (throughput)

▲

jbellis 5 hours ago | parent | next [-]

As far as benchmarketing goes they clearly went with prefill because it's much easier for apple to improve prefill numbers (flops-dominated) than decode (bandwidth-dominated, at least for local inference); M5 unified memory bandwidth is only about 10% better than the M4.

▲

GeekyBear 5 hours ago | parent | prev | next [-]

In previous generations, throughout was excellent for an integrated GPU, but the time to first token was lacking.

▲

danudey 5 hours ago | parent [-]

So throughput was already good but TTFT was the metric that needed more improvement?

	▲	zamadatix 4 hours ago \| parent \| next [-]
		To add to the sibling "good is relative" it also depends what you're running, not just your relative tolerances of what good is. E.g. in a MoE the decode speedup means the speed of prompt processing delay is more noticeable for the same size model in RAM.
	▲	convenwis 5 hours ago \| parent \| prev [-]
		Good is relative but first token was clearly the biggest limitation.

▲

case540 5 hours ago | parent | prev | next [-]

I assume it’s time to first output token so it’s basically throughput. How fast can it output 8001 tokens

▲

fragmede 5 hours ago | parent | prev [-]

No you don't. Not as a sticky mushy human with emotions watching tokens drip in. There's a lot of feeling and emotion not backed by hard facts and data going around, and most people would rather see something happening even if it takes longer overall. Hence spinner.gif, that doesn't actually remotely do a damned thing, but it gives users reassurance that they're waiting for something good. So human psychology makes time to first token an important metric to look at, although it's not the only one.

	▲	MrDrMcCoy 5 hours ago \| parent [-]
		Some kinds of spinners serve as a coal-mine canary indicating if the app has gotten wedged. Not hugely useful, but also not entirely useless.