I don’t think that’s considered secure enough, see the other answers and the push for passkeys.

I mean, if passphrases were good for anything you’d directly use them for the ssh connection? :)

Passphrases, when strong enough, are fine when they are not traversing a medium that can be observed by a third party. They're not recommended for authenticating a secure connection over a network, but they’re fine for unlocking a much longer secret that cannot be cracked via guessing, rainbow tables, or other well known means. Hell, most people unlock their phones with a 4 digit passcode, and their computers with a passphrase.