| ▲ | dofm 5 hours ago | |
The issue is not RAM size. It's memory bandwidth! The 12B and 31B models will be useless on a Pi 5; maybe the 12B can be persuaded to run, but it may not manage more than one token per second. It only manages 17GB/s memory bandwidth (I have seen a suggestion that the 4GB model manages more). But yes — some sort of small reasoning-oriented model (Ornith?) seems a better candidate than Qwen 35B. (Don't get me wrong, I think the 35B model is ace… just seems like at least an unusual choice here) | ||