| ▲ | walrus01 6 hours ago | |
I assume you mean putting only the 32B active parameters on the GPU, and the rest on a bunch of regular server DRAM like on a 768GB to 1024GB RAM server? Because Kimi K2.6 in Q4 is about 584GB GGUF size on disk and will use slightly more than that in RAM, Q8 is 595GB. | ||