Would BRAM even have enough bandwidth? The reason I quoted logic cells is because that's the way to get instant throughput, which is practically the only reason to use an FPGA over something like a TPU.

I think so, because memory bandwidth really comes from bus width more than clock speed. You can construct 36-bit wide BRAM arrays with bus width comparable to HBM, just by specifying multiple BRAM arrays in parallel.

Never tried anything like that, though.