I find 10 to be very usable. It’s not (that) interactive but it chews through tasks. I let Kimi churn away at 4 overnight and it gives good results that are ready for me in the morning.

Yes, I thought when I was starting that 1u and 2u form factors were to save space. Maybe they are, but they also have the advantage of moving air front to back very effectively through and over the components. Though there still must be some need because I see even those boxes have optional manufacturer built memory shrouds to try to force airflow between the DIMMs.

I had some 120x38mm fans from another server box that I pulled out because they were too loud and I didn't need the static pressure they were giving. They went in here. That 13mm (and the extra 1k rpm) moves so much more air.

I built a dual epyc server with 64 cores and 1 TB of DDR4. Draws around 800W or so under load. I used off the shelf liquid cooling. It is audible but not noisy.

The trick is to turn on the cooler's RGB in your 6000€ server to get a free speed boost. I am not liable for sysadmin's heart attack upon reading this.

One thing I'd like to try is MoE offloading: I have 2x32 GiB of VRAM and 128 GiB of DDR5 running at 4800 MT/s (only 2 channels though). I've seen people post difficult to believe MoE offloading results albeit a decently long time ago with older models. Maybe there is a quant that would fit with MoE offloading?

That said, I am guessing my problem is not enough RAM - but this poor consumer platform struggles to do memory training with 128 GiB as it is.

Now I surely regret not having gotten Threadripper and 256 GiB of RAM in the before-times.

With a 4-bit quant of GLM-5.2, I can get about 0.8-1.1 tok/s on an underclocked dual Xeon E5-2698 v4 with 512GiB of DDR4-2400. I think it was specifically a Q4_K_M quant. Of course, the time-to-first-token is absolutely atrocious.

Which is completely insane for a ten year old configuration.