"flag" and "downvote" are two different tools with two different purposes.

"downvote" seems more appropriate for for "this is not interesting and should be less prominent".

"flag" seems more appropriate for "this should not be here at all".

By way of an example, on a political story, if you say something merely unpopular, you'll get downvotes and replies; if you say something hateful, you'll (usually) get flagged.

I agree with you, but that's not what happens for polarizing topics that are technical in nature and not political. People on HN seem to flag comments rather than downvote them.