I saw Musk discussing that and his main argument in favour is there's unlimited solar power whereas on earth there are a lot of power constraints.

Sure, in the same way that there is unlimited solar power on earth.

There are orbits with 24/7 or near-24/7 sunlight - but those are very undesirable if you want a low-latency data link back to Earth. Just like you can get 24/7 sunlight on the North / South Pole - but they are still pretty bad locations for a data center.

LEO orbits like those used by Starlink have far better connectivity, but about the same sunlight exposure as the surface as the planet will be between you and the sun about half the time.

Also, power is the easy part. Cooling is far harder.