This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.

However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.

▲

johntb86 18 hours ago | parent [-]

Maybe if it's worse than next-token-prediction it could still be used as a drafting model, similar to how DFlash works.

▲

cubefox 18 hours ago | parent [-]

What's that?

▲

johntb86 17 hours ago | parent [-]

https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.

	▲	cubefox 7 hours ago \| parent [-]
		I'm pretty sure fast speculative decoding models need to have the same objective (NTP, MDM, NCM) as the larger model they try to accelerate.