| ▲ | amluto 3 hours ago | |
> Speculative decoding is not really worthwhile for sparsely-loaded models. If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss. Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance. | ||
| ▲ | johndough 6 minutes ago | parent [-] | |
> If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens You can! > AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation. https://machinelearning.apple.com/research/introducing-third... | ||