You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.

Ahh, I remember hearing that before, and it makes general sense too.