At the same time we're at a point where synthesizing your voice is getting more trivial everyday, you need only a few seconds of it and you can be made to say whatever someone wants.

Sure, but that doesn’t mean they learn everything I said: Passwords, personal details etc.

Also, getting a voice sample in the first place gets significantly easier that way: Not everybody publishes video or audio recordings of themselves online.