Note that non-free firmware in a network card, for example, doesn't affect anything, if the traffic is encrypted (and ideally routed through VPN so that the card has no direct Internet connection). So in some cases we can isolate non-free components so that they cannot do any harm. Modem in a phone, probably can be isolated also.

A non-free firmware in a network card can

1) deny you service at the will of its true owner,

2) stop receiving updates rendering this piece of hardware dangerous to use with no recourse.