| ▲ | alexchantavy 2 hours ago | |
How many tok/s are you getting? What gen mbp? | ||
| ▲ | mattdw 23 minutes ago | parent | next [-] | |
My M4 Pro 48GB gets about 13tok/s, in both 3.6 and 3.8 27b Qwens. Qwen A3B and Gemma get closer to 100tok/s from memory but the results are pretty poor for coding tasks. Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed. | ||
| ▲ | MrScruff 20 minutes ago | parent | prev | next [-] | |
I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb. | ||
| ▲ | dominotw 44 minutes ago | parent | prev [-] | |
i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two. | ||