diffusion gemma?

SillyUsername 18 hours ago | parent | next [-]

It pretty bad. Just ask it to quote something like the first paragraph of moby dick (i.e. "call me ishmael...") it barely gets past 5 words before it completely gets it wrong. Even the worst normal 4B models get 3-5 sentences.

	▲	gessha 16 hours ago \| parent [-]
		That’s because diffusion models struggle with memorizing sequences while it’s essentially what autoregressive models excel at. I’ve tried to fine tune a diffusion model to memorize an MD5 hash and after days of hacking on it, it failed to completely memorize it. Autoregressive models of any size learn it over <1000 iterations.

▲

UncleOxidant 20 hours ago | parent | prev [-]

Was mostly an experiment. You're not going to use it for any real coding. Hopefully we'll see another updated diffusion gemma in the future.