It's not 'downvote abuse' if it's working exactly as intended. The community decides what's 'perfectly fine and neutral.' If your comments follow the guidelines, at least they won't get deleted.