Remix.run Logo
▲ kimixa 11 hours ago

I think the major difference is they have a set of profile data they use to manage the placement of expert weights.

"Hot" commonly used experts are kept on the GPU, and "cold" less used experts are bumped to system ram and calculated on the CPU. So it can run larger than vram sized models without completely crippling performance.

I'm not exactly sure what the changes they made to the architecture are for, perhaps they're aimed at extending the difference between the "hot" and "cold" expert usage?