Yep, the only thing you need to do is to disable password auth. They aren't going to be able to crack your SSH key no matter how many times they try.