smart people are born into dumb families pretty regularly

Are you implying widespread infidelity here, or are you making the case that something besides "nature" may be determining intelligence?

I did edit that out before your comment because I didn’t feel I needed it, but I still stand by it.

There is still quite a lot of randomness in genes, the idea that intelligence would always be the average of the parents would require that that a very large number of SNPs are involved. GWAS studies do say this but this is more a side effect of using linear regression for the scores as this assumes independence which I think is not a safe assumption. I think some intelligence genes can be recessive so you can have two carrier parents where 1/4 of their children will be smarter than either of them.

I should also add, by pretty regularly I mean from the point of view from the smart people. Given a sample of smart people how often are they notably smarter than both parents.