That’d break a lot of GPU setups

Only if you want to virtualize it or have vms, for VBS it simply disables hardware pcie memory space isolation. (With IOMMU on, each pcie device gets an isolated memory buffer).