You know what else is heritable? Wealth, the possession of which tends to help with standardized test scores.

That causality goes in the other direction.

Wealthier parents tend to be smarter (that's how they got wealthy or managed to keep inherited wealth) and tend to have smarter kids... who then tend to up on the wealthier side of the spectrum.

It's very unfair. It's also very real. Your fantasy is not real.