That is exactly wrong. The measure of heritability used in the scientific literature is very much tied to genetics, just not in a very direct way. That is, heritability is a measure of how much of the variance in a trait is explained by genetics vs environment. In this sense, wealth will have a relatively low heritability, because it is weekly tied to genetics, even though it is very much a trait most people inherit from their parents. Skin color will have a high heritability, because the variance in skin color is almost entirely explained by genetics.

The unintuitive part is that traits with almost no genetic variance at all, such as the number of arms, have very low heritability - since, in a population study, almost the entire variance in the number of arms will be explained by environmental factors (very very few families have 1 or 3 arms as a recurring trait - and there are way more people who lose their arms during life).