How else are you supposed to do identify verification over the phone?

I think if the war against phishing online has taught us anything, it's that humans can't be trusted to not reveal secrets to scammers. Only machine-to-machine public key authentication (like TLS or WebAuthn or U2F) is truly phish-proof.