For one, because it's an indicator of health and healthy development, which are in turn related to other desirable features. Analogous things can be said for women.

You also can't ignore assortive mating. We're not all in the same mating pool, as it were. Women prefer men taller than they are, but that doesn't mean that a woman of 1.25m will pair with a man of 2m. The typical height difference between husband and wife is about 10-15cm, regardless of the absolute height of either spouse.