That's what key signing parties are for. In person verification.