I still don't understand why Google did effectively nothing with their Gemini Diffusion demo from last year.

It was really cool, and really fast, and just went quiet.

They did release a Diffusion Gemma a few months ago but I've not seen much of a splash from that one, either.

It's not Googles fault. It's our fault, llama.cpp doesn't support it, vllm can't really offload to host ram and while huggingface transformers supports it, it only can do f16 quant for MoEs like Diffusion Gemma. Unusable for low spec machines where diffusion really could speed up things.

We're sleeping on a amazing model right in front of our eyes. With diffusion models you can properly ban phrases. You can decensor and deslopiffy the model output without touching a single weight or doing a single backwards pass.

Their preprint showed it worked well with small batch size, but most prod deployments right now have rather large batch sizes from serving multiple users

Difussion Gemma performed a lot worse than its equivalent autoregressive counterpart (it was particularly bad because rather than a sort of constant-ish gap you might see between 2 model classes, the gap seemed to worsen the harder the task/benchmark) and the speed benefits got negated at scale. Not really surprising we haven't heard much.

This talk serves as sort on update on where they are with it: https://youtu.be/r305-aQTaU0?si=j3IqG9BAT0uumPeA

At a high level: It doesn't benefit from batching to reduce memory bandwidth in the same way that a "standard" autoregressive model does, which ends up meaning that it's great at reducing latency for individual/low-concurrency users and delivers better outcomes on certain tasks but much more expensive per token at scale.

They probably did not get it to work well?

Good at research, terrible at product?