A GPU can be used for inference, but, for that use, there are much better choices. Apple designed their NPUs for that, IBM added an NPU to their mainframe chip and AMD and Intel are planning on adding inference-specific instructions to the amd64 ISA.