It happens all the time, and its as easy as sending a phone a text, or a packet, or escaping a sandbox, but you'll rarely be aware of it when you're infected because unlike the old days where malware would fill your screen with ads or something today they just silently collect your data or use your internet connection for careful port scans or DDoS attacks. NSO Group spyware (or similar) could be on your phone right now.

Hell, cellphones these days ship with spyware pre-installed. Samsung being the one of the worst for filling their phones with their own apps which spy on you constantly.