It happens already. These are plenty of cheap V100s on eBay, and PCIE to SXM2 adapters

External example: https://ebay.io/m/lV8UsD

Internal example: https://ebay.io/m/z1ygRU

V100s are three generations behind current and missing many of the features that modern inference benefits from, but they are the cheapest way to get a 32GB gpu.