MoE has nothing to do with multimodality.

MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.

Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.

As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171

Wait I thought the router ends up specializing the experts?

Like there is no explicit goal aside from each 'expert' getting roughly equal weight?

And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?

I do not know what I am talking about, this is my limited understanding...

> You can think of it like sharing.

was this meant to read "sharding"?