Remix.run Logo
ComputerGuru 4 hours ago

Do infra providers reveal that level of implementation detail?

scrlk 4 hours ago | parent [-]

I've seen a few articles from providers talking about KV cache quantisation, but it's not something they explicitly point out like they do with weights.

So you could end up paying more for unquantised weights, only to get silently hit with a quantised KV cache...