OK that’s what I figured you meant. FWIW, MoE as a term of art means something different, what I described. It’s internal to a single model, part of the logit generation process.

	▲	measurablefunc 4 days ago \| parent [-]
		That's fine, you can pretend my entire diagram is one NN, end result will still be the same whether you put it all inside one box or break it out into many.