> I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.

Taalas needed a giant chip (6nm) for an 8B model.

At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.

▲

greenknight 2 hours ago | parent [-]

Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

▲

cherioo 2 hours ago | parent [-]

Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all.

The more problem like these they solve the more they will look like GPU.

▲

nextaccountic an hour ago | parent [-]

couldn't one just add some hundreds of GB of HBM?

	▲	kimixa 15 minutes ago \| parent [-]
		Yeah, but then there's the size of KV cache needing to be read through that HBM interface for each token, putting a hard limit on the tok/s based on the memory bandwidth. On some models a large context can be a notable proportion of the size of the weights themselves. For example, qwen 3.8 27b uses ~64kb/token for the kv cache - so for a 256k token context that's ~16gb of the kv cache for a ~54gb model (assuming 2 bytes-per-param/f16 for both). So if the current non-baked-in chip is already memory bandwidth bound, as is often the case for current hardware and models, and the "only KV cache in HBM" chip has the same total memory bandwidth, it can only ever be (54/16)=~3.4x faster for the baked in-silicon model. EDIT: I guess actually (54+16)/16=~4.3x faster, as the current implementation would need to read that KV cache too :)