Wow this is really impressive. Throwing away the VAE feels counterintuitive - I'm shocked it works so well! Appreciate that you shared your training insights on the blog post!

Reminds me of this great paper where they trained a LLM that beats LLaMA2-13B from scratch with < $0.1M: https://arxiv.org/abs/2404.07413

Does it? Text generation models have been decoder-only for a while, with great benefits to their training. It's surprising to me that more attempts weren't made for other modalities.

$100k seems like a lot. I guess it isn't in the grand scheme of things but still