Problem is this argument can be leveraged to wipe out any living or non-living thing whos numbers pose a potential threat. Other religious groups, races, even sufficiently different cultures.

Who killed the Neanderthals? Were sapiens actually smarter or were they just less accepting of those different than them?