I was peripherally looking into this for a similar problem domain: https://en.wikipedia.org/wiki/Private_set_intersection

Related to Zero Knowledge Proofs, the advantage is that phone numbers need never be shared in cleartext, preempting whole classes of attacks. However, could be overkill for your needs, and I am not sure how well current techniques would scale.