> the “smart” aren’t necessarily having fewer kids than the “idiots.”

They are: https://en.wikipedia.org/wiki/Fertility_and_intelligence

There are some confounding factors, like education and income also affecting fertility, and correlated with most measures of intelligence, but that doesn't actually affect the diagnosis - regardless of if intelligence directly lowers fertility, or if the effect is mediated by e.g. education, the net effect is that smarter people have fewer children (in our current environment).