My understanding of the average age of a human being 30s-40s is that the average was skewed downwards heavily due to high infant/child mortality rates

AFAIK at birth the average life expectancy was even slightly lower (30-35). At 15y it was about an extra 30-40y.

Whatever the exact numbers: definitely skewed, like you said.