I'm not an expert on this, but this sounds a lot like a larger-scale MoE (Mixture of Experts) type of architecture.

As I understand it, in an MoE model, you essentially have hundreds of smaller sub-models ("experts") that are good at different tasks, and for every generated token, a single "master" model chooses which ones are most relevant to participate, and you only activate them.

▲

janalsncm 3 hours ago | parent [-]

In MoE systems the routing decision is made per-token, not per prompt or task. It’s one of ML’s many confusing naming conventions.

Even more confusingly, there are older pre-LLM MoE systems which ensemble and pool the predictions from multiple sub-components. For example in a random forest you could take the majority vote of the decision trees or the average of their numerical predictions.

After that, we developed neural net architectures for predicting a single thing like whether the user will click on your ad. An MMoE is in the same family.

And so now we are at massive MoE networks for LLMs which have similarities with MMoE in that the “decision” is about the very next token to predict.

▲

lukan 2 hours ago | parent [-]

"In MoE systems the routing decision is made per-token, not per prompt or task."

Have there been experiments with doing it per task? Like, "oh this is python project, use this model" "oh this is about writing fantasy, use this"?

	▲	janalsncm an hour ago \| parent [-]
		It’s a good idea. The results probably depend a lot on how close your task is to the benchmarks though. I think OpenAI already has (had?) a feature like this called “auto” mode for thinking.