> The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090

Without leveraging system RAM and/or SSDs, I don't think you can, or how exactly are you running this, if this is something you are doing today? With CPU/expert offloading you could probably do it with a 5090 + 1TB of RAM or something like that, but absolutely not on a single 5090 entirely within VRAM.

▲

robotswantdata 6 hours ago | parent | next [-]

Yes hybrid approaches are much better than people realise.

There are a lot of optimisations that are not in the public sphere, source working on start up in this space

	▲	embedding-shape 5 hours ago \| parent [-]
		> There are a lot of optimisations that are not in the public sphere Sure, but if we're participating in public discussions, isn't it more fun if we talk about things people can actually read and understand, rather than secret stuff other's can say work, but no can actually validate or know how it works? It sounds like "hybrid approaches are much better than the public is aware, because everything else is private and secret", but also: ok, so what? No one can run that anyways, (yet?), so why it matters?

▲

rhdunn 6 hours ago | parent | prev [-]

Yes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.

	▲	embedding-shape 5 hours ago \| parent [-]
		Alright, I guess I misunderstood. To be fair, this part: > The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. Is painting a very different perspective, even considering the latter parts it's hard to read that as "Of course offloading everything else that doesn't fit on the GPU itself". But anyways, it's been clarified now so no harm :)