Solar panels can in principle be made very thin, since there are semiconductors (like CdTe) where the absorption length of a photon is < 1 micron. Shielding against solar wind particles doesn't need much thickness (also < 1 micron).

So maybe if we had such PV, we could make huge gossamer-thin arrays that don't have much mass, then use the power from these arrays to pump waste heat up to higher temperature so the radiators could be smaller.

The enabling technology here would be those very low mass PV arrays. These would also be very useful for solar-electric spacecraft, driving ion or plasma engines.