I've been watching the drizzle of LLM papers come through, and I think we're going to hit a 1T param MoE on consumer hardware before this year is out. It'll still be behind the bigco models, but it'll be a force multiplier. Ideally, we'd get these models to run on a CPU. MS BitNet is one way to do this. You can already run ternary LLMs on consumer CPUs with a decent tps.

▲

mattmanser 2 hours ago | parent [-]

Though what is consumer hardware right now?

Can we still classify 5090s as consumer hardware given how expensive they are? They're £3k at the moment, and it looks like it's only going to get worse unless the AI bubble pops.

	▲	all2 2 hours ago \| parent [-]
		I was thinking more in terms of 24GB of VRAM total. I started sketching the architecture for such a model this afternoon, nothing novel, just combining existing advancements in the field. It looks achievable.