Cool — but is that model really the right choice for the task?

I guess it is only 3B active which helps a lot but is Gemma 4 E4B not more practical?

aamargulies 4 hours ago | parent | next [-]

Qwen3.5-4B would be a good (better?) candidate. It uses a gated, deltanet hybrid, so your KV cache stays nearly flat as context grows, important for RAM-constrained environments like the Pi.

▲

0xbadcafebee 6 hours ago | parent | prev [-]

Qwen 4B and 9B should be faster and better reasoning than Gemma 4 E4B. Other good options for that much RAM are Gemma 4 12b and 31b. Gemma 4 E4B would be better for native audio, but OP is using Whisper for STT so prob doesn't matter

	▲	dofm 5 hours ago \| parent [-]
		The issue is not RAM size. It's memory bandwidth! The 12B and 31B models will be useless on a Pi 5; maybe the 12B can be persuaded to run, but it may not manage more than one token per second. It only manages 17GB/s memory bandwidth (I have seen a suggestion that the 4GB model manages more). But yes — some sort of small reasoning-oriented model (Ornith?) seems a better candidate than Qwen 35B. (Don't get me wrong, I think the 35B model is ace… just seems like at least an unusual choice here)