Not getting into how we define what are valid complaints vs invalid, because that's besides the point.

You'd find it exhausting. People similarly find the race and gender preoccupation exhausting. Your justification for the different treatment seems to be "for these categories, people are obligated to not find them exhausting, but not for other categories." But that's not well-motivated.

In a height-preoccupied literary world, can you imagine objecting to the height preoccupation? If you did, how could you justify it, if it's not a defining part of your existence?