I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.

I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.

It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)

What rumtime are you using?