Have you thought about making the whole thing "self-similar"? Every time I hear about MoE I think (and I know it's way easier thought than done) "why stay shallow"? I mean by that: would it be possible to extend/adapt the architecture so that an expert can be a previously trained Mini-AGI model? And recurse like this? Inuitively I would think some form of generalization could happen, as higher level experts (in the recurrence stack) would become sort of the "intuition" layer.

Making model to consists of many small modules is inefficient on GPU, especially as routing adds data dependencies, etc, and especially with pytorch (compared to a custom kernel).

The difference might be smaller on a CPU which has limited parallelism.

But it's basically equivalent to a very deep model which might be problematic for training.