It should just about be usable in 32GB.

On a consumer hardware it would be nicer. With no GPU/iGPU or a 6-8GB VRAM.

	▲	hnfong 2 hours ago \| parent [-]
		It would be somewhat slow on a CPU only machine, but it still works. Besides, Macbooks with 32GB RAM is consumer hardware, just maybe on the higher end.

npodbielski 4 hours ago | parent | prev | next [-]

It is. I am running it on R9700

bakraman 2 hours ago | parent | prev [-]

RAM is never the issue, it's always the compute power

	▲	CamouflagedKiwi 12 minutes ago \| parent \| next [-]
		It's absolutely not for these models. There are plenty of consumer GPUs out there with 8 or 12GB VRAM - they are comparatively very fast at inference but just aren't big enough to run lots of the models you want. Also context management is a massive pain.
	▲	spider-mario an hour ago \| parent \| prev \| next [-]
		RAM is not “never” the issue. My iPhone and MacBook Air could both run larger and more capable models if they had more RAM.
	▲	tuetuopay an hour ago \| parent \| prev \| next [-]
		Quite the opposite, RAM is always the issue. More specifically, high bandwidth RAM.
	▲	geek_at 2 hours ago \| parent \| prev \| next [-]
		and memory bandwidth
	▲	mhaberl an hour ago \| parent \| prev [-]
		what??? not true! for inference the compute is the last thing we need more of. memory bandwidth is the numebr one blocker, after that the inefficiencies that where introduced with MoE models (and all new large models are made that way) Here is a quick read: https://news.ycombinator.com/item?id=49324600