No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60).

I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.

I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.

The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.

Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.

> about $80 (£60)

Man, I wish I lived where you guys lived.

How the hell did you fit 6 P4s in a mATX case?