Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.

One appeal of it is for RL. If it ends up being a lot faster for generation, you'll be able to do a lot more RL.

If people can make RL scalable-- make it so that RL isn't just a final phase, but something which is as big as the fine tuning, then diffusion models are going to have an advantage.

If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actually refine their outputs, so we're not talking about some kind of refinement along the lines of: initial idea -> better idea -> something actually sound.

Feels like the sodium ion battery vs lithium ion battery thing, where there are theoretical benefits of one but the other has such a head start on commercialization that it'll take a long time to catch up.