I don't thinnk just MoE will solve it. If you hit constantly different expert layers, you can't outsource layers efficently and have to swap it in.

MoE will be faster because it will read less memory for sure, you still have to have it though.