We don't poll all of the people. We exclude everybody below 18, for example. We exclude a lot of people with criminal records. We use a multi-stage representative democracy, such that large numbers of voters basically don't count at all. (And that's before we go out of our way to draw districts specifically to exclude people).

To the degree that democracy works, it works as you say. But we systematically fail to include all of the people, so the dumbs often don't cancel out.