Remix.run Logo
▲ jacquesm 5 hours ago

If you increase the power limit on the card you will be able to get much more mileage out of this modification. At 450W the difference is much, much larger but that is not something I'm comfortable with doing for longer runs. Note that the stock settings are already pretty extreme for that cooler, and that ambient temp and free airflow are going to be impacting the resulting GPU temp in a very direct way. I have my 3090's in an external box (with its own separate PCIe backplane) so they can talk p2p and have 'fresh air' rather than in the enclosure.

▲unsnap_biceps 4 hours ago | parent | next [-]

Is your external box something you put together or is it a product you purchased?

▲jacquesm 3 hours ago | parent [-]

I put it together myself. It was a lot of work and I learned a lot of nasty lessons along the way. The short version of the recipe:

- get the largest PSU you can find (1500W at a minimum, more is better)

- get the ADT PLX Quad PCIe backplane. Run 4 thin (single slot) GPUs inside your main enclosure, stick the ADT interface board in between two of the thin boards (don't stack them close to each other or they won't be able to breathe).

- ensure you have as much or more RAM as you do VRAM

- Ensure your CPU has enough PCIe lanes to run at least five boards in x16 (four for the GPUs in the main enclosure, another for the PLX PCIe switch)

- In the BIOS set BAR1 to 'resizable' and > 4G decoding should be enabled.

- ditch the cables that come with the ADT board, they may work but there are much better ones out there and they're not expensive, this will save you days of troubleshooting

- Build your own enclosure rather than trying to adapt a standard one, it won't work anyway (the board pitch is wrong, as is the placement of the mounting holes).

- Put another four double or triple boards on the backplane.

- Add another 3x 140 mm fan to the front of the case so you get good airflow without the feeling of being next to a Harrier at take-off

- I used 3D printed parts to mount the PSU and the board as well as the fans (see below).

- Make sure the board is properly grounded to the enclosure

- If you're running 3090's or other cards that are P2P capable that were hobbled by NV then look at this https://github.com/aikitoria/open-gpu-kernel-modules driver module.

- Power on the ADT board, wait three seconds then power on the main machine. Enjoy your VRAM :)

If you have more questions please email me and I'll be happy to help out.

▲Tiberium 5 hours ago | parent | prev [-]

Yes, but you can get up to a 40-50% increase in some workloads even with the same power limit, and some cards have it locked anyway. I just felt like it was a litle misleading for you to mention the power in your original post, since the unlock works regardless of that.

▲jacquesm 5 hours ago | parent [-]

Interesting, I have 7 3090's here of different plumage and they all require a power increase to get > 25% gain with this modification, would you mind sharing the type of card you have and how it is cooled?

Obviously, if there are 3090's out there that respond differently then I will be happy to retract that.