(I'm a PP-ASEL, earned in and around SF Bay Area air spaces almost three decades ago)

First, people are more likely to crash during takeoffs and landings, not in the middle of a flight. Second, it's also not just about the damage caused to others, and even if you crash in the middle of nowhere quite a few people will be busy dealing with it - in that case, remoteness is not a good thing but actually more costly. They won't just leave you and the airplane lying at the crash site until nature reclaims it all.

Furthermore, it is not just the flying itself. If that is what you wanted, the crazy early times of flight would have been for you, when everybody could just do whatever they wanted, wherever they wanted. Much of the learning for the license is the rules and procedures. Air space if very busy! I have small class D airports where there is so much traffic sometimes that I can hardly get a word in to announce myself. This will have the potential for stress and errors long after you got used to the flying itself.

I can't even begin the shortcuts someone's thinking has to take to think flying and driving should be treated the same.