Whatever the differences, they would have been attractive enough to Homo Sapiens to breed with.

Maybe not willingly, though. Look up Danny Vendramini's neanderthal predation theory, and consider that modern X chromosomes carry no neanderthal DNA, indicating that all interbreeding involved neanderthal males and human females.