Some epigenetic effects are semi-heritable too, eg maternal exposure can be transmitted. That's in addition to environmental effects like you mentioned. Two otherwise identical cohorts can inherit the same genetic predisposition for depression where one manifests and another does not entirely due to their circumstances.

Evolution is just super super messy.