If that were true then we would expect to see a positive correlation between income and family size, but households making 500k are basically the same size as households making 50k.

Your specific claim may indeed be true, but it's misleading. The relationship between income and children is U-shaped. From middle incomes to higher incomes, fertility rises. It is also important to point out that income is tied to other factors in America. You're going to disproportionately find your $500k earners in a handful of superstar coastal cities. Those things need to be controlled for if you want to isolate the effect of income on family size.