I feel like what we really need is a GPU "socket" like we have for CPU's. And then a set of RAM slots dedicated to that GPU socket (or unified RAM shared between CPU and GPU)

Unfortunately socketed processors only really work with DDRx type DRAM interfaces. GPUs use GDDR and HBM interfaces, which are not ideal for sockets. In the case of HBM, you have 1024 data traces per DRAM chip, which would make the socket have an insane number of pins. GDDR has fewer pins, but makes up for it with higher data rates (32 pins at 16Gb/s) and that is impractical to use in a socket due to the variance in contact area resulting in impedance matching issues.

HBM would most likely have to be integrated directly on the GPU module given performance demands and signal constraints.