Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server.

If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.

(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)

Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.