| ▲ | cyberax a day ago | |
I'm running qwen3.8-flash-next at 4-bit quantization on two AMD Radeon AI PRO R9700 cards at 75 tok/s. And I can actually run two models in parallel at around 50 tok/s, so you can get by with only one card. I switched from NVidia to AMD for a fully open-source stack for my local inference. AMD Radeon AI PRO R9700 is $1600 right now (I bought them last year for $1300). They are also quite power-efficient. My setup typically uses around ~300W during inference. | ||