Four of these together should, in the abstract, let you run 200GB models, which is where things get very, very interesting.

The biggest Deepseek V2 models would just fit, as would some of the giant Meta open source models. Those have rather pleasant performance.

In theory, how feasible is that?

I feel like the software stack might be like a Jenga tower. And PCIe limitations might hit pretty hard.