I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that inference is bound by reading the weights, so stop fetching them from far away. I wanted to see how far that idea stretches on a 'consumer hardware': every weight resident in SRAM, zero DRAM My chip only gives you ~3 MB to live in, so the model is 3.16M parameters and the tokens are characters, but the physics is the same bet. Only difference is their model is actually good!

The annoying part: I wanted this live last week, which would have landed the same day as the AMD acquisition news, on AMD's own silicon no less. Life got in the way, so here it is a week late instead.