The main danger is a virus that infects everybody's phones and then takes control of the telephony modem, e.g. like a DDOS attack.

That's why you can't have root access to the modem even though you technically own it.