> whereas fossil fuel infrastructure is built in places you'd never go to voluntarily.

IMO that's the consequence, not the cause, of the infrastructure: https://en.wikipedia.org/wiki/Hambach_surface_mine

Or sometimes the consequence of the closure of that infrastructure with no transitional plan: https://en.wikipedia.org/wiki/Merthyr_Tydfil

The latter is absolutely worth caring about even for the greenest of green energy. Even just PV on rooftops, when a town becomes a ghost town.