Not a perfect defense, but sufficient to make your key much harder to exploit: Use a Yubikey (or similar) resident SSH key, with the Yubikey configured to require a touch for each authentication request.