Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...

▲

josu an hour ago | parent [-]

I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?

	▲	l33tman an hour ago \| parent [-]
		Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?