Well yes but this is also a feature: you can store passwords without having the private key available. That can be handy.

I don't really see the value in an attacker being able to store new passwords anyway. Besides, in order to do that they'd already have to have breached my private git server too.