I wouldn't bet on hardware flooding the market. I bet the machines running in the data centers don't use traditional PCIe connectors and cards. Maybe somebody could pull the chips and put them on standardized PCIe cards, but that is not a given.

It happens already. These are plenty of cheap V100s on eBay, and PCIE to SXM2 adapters

External example: https://ebay.io/m/lV8UsD

Internal example: https://ebay.io/m/z1ygRU

V100s are three generations behind current and missing many of the features that modern inference benefits from, but they are the cheapest way to get a 32GB gpu.