Remix.run Logo
▲ julianlam an hour ago

Something doesn't seem right. Vanilla llama.cpp can hit 50 t/s on an R9700. Q6 (albeit with limited context of 64k, more if kv cache is quantized) fits entirely on the card.