> Even if the output is like 5-6 tok/s

On a 3T model I’d imagine you’d be closer to 0.05 tks

Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.

▲

zozbot234 3 hours ago | parent [-]

Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.

	▲	amluto 2 hours ago \| parent [-]
		> Speculative decoding is not really worthwhile for sparsely-loaded models. If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss. Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.