From what I’ve read, there was a lot of thought about them having fallen from their ancestors. Some of the eugenics types wrote about this as a cautionary tale about mixing with other races or letting them share power. The reasoning only makes sense if you start with the conclusion and work backwards trying to make it fit.