No, you can't divide the entire size by the expert count. A lot of weights are constant for all tokens, so total active count is ((2800-(shared)/896)*16 + (shared))

▲

msdz a day ago | parent [-]

TIL, that makes a lot of sense, and thanks for the correction.

	▲	HarHarVeryFunny a day ago \| parent [-]
		Just to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream).