I think it's mostly fear that bad actors will make government take that little bit of freedom that they are enjoying

Eh, it1.'s not really fear.

The spectrum is fragile. A few people being assholes can mess it up for thousands or millions of people with a few hundred bucks of equipment.

While you or me may not cause a mess, there are millions and millions of people around and eventually you hit the random number generator where people fuck things up.