| ▲ | ThunderSizzle 6 hours ago | |
An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions. | ||
| ▲ | syntaxing 2 hours ago | parent [-] | |
Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine. | ||