If your need is security then really that should be based on in person trust.

Or at least via a proxy.

So contact invitation can just be handled with use-once codes (or at least trivially burnable ones).