In MoE systems the routing decision is made per-token, not per prompt or task. It’s one of ML’s many confusing naming conventions.

Even more confusingly, there are older pre-LLM MoE systems which ensemble and pool the predictions from multiple sub-components. For example in a random forest you could take the majority vote of the decision trees or the average of their numerical predictions.

After that, we developed neural net architectures for predicting a single thing like whether the user will click on your ad. An MMoE is in the same family.

And so now we are at massive MoE networks for LLMs which have similarities with MMoE in that the “decision” is about the very next token to predict.

"In MoE systems the routing decision is made per-token, not per prompt or task."

Have there been experiments with doing it per task? Like, "oh this is python project, use this model" "oh this is about writing fantasy, use this"?