| ▲ | smcleod 4 hours ago | |||||||
50-70tk/s is what I get on my m5 max on a 5-6bit Qwen 3.8 27B? | ||||||||
| ▲ | Casteil 4 hours ago | parent | next [-] | |||||||
I don't know what black magic you're up to but I see more like 30-35t/s on a 16" M5 Max using 3.8:27b Q4, regardless of whether it's mlx or gguf. qwen3.5:122b-a10b is significantly faster at around 60-65. | ||||||||
| ||||||||
| ▲ | sscaryterry 3 hours ago | parent | prev [-] | |||||||
I tried 8-bit, perhaps I should try 6-bit. | ||||||||