Because of the reciprocity of absorption and emission: the panels which absorb more also proportionately radiate heat back into space at night more effectively than whatever they cover (which presumably has a higher albedo)

You might then wonder why the albedo of the poles matters (why are scientists concerned about that?). The answer to that is that the poles are very cold and black bodies radiate energy as T^4.

So you don’t want to have low albedo where it is cold and a high albedo where it is hot because then the most emissive surface isn’t emitting much heat due to low T and the places with high T have low emissivity. Obviously heat transport within the atmosphere is hugely important for this analysis, which you can’t easily approximate with napkin math.

However solar panels are most likely to be placed where it is relatively hot (because humans mostly don’t live at the poles). So that actually good because now you have more emissive surfaces with high T, which promotes global cooling.

That's an impressively thorough answer to a vague wonder. Thanks