You should give it longer than an hour before you start complaining about downvotes. Or just let your comment stand on it's own.

Seconded. You can't see all the up and down votes, only the balance at the moment you look, and it's not too uncommon to be negative or even dead and be upped or vouched back to life later.