The message "guys, it's really not that smart to let in a huge number of people from a completely different culture, this is going to be a problem" was completely ignored and labeled racist.

Yes, it's probably the left's single largest mistake in messaging in a hundred years. It's understandable, since so much of the right's take on this topic is in fact often more racist than pragmatic, but that's also a "loudest voice" problem, and the left has those voices too. I don't know how a country with the US's history and culture can even talk about this without those voices dominating the perceptual bandwidth. It's essentially impossible, I guess. And being ignored pushes more people into that tone. This is the vicious cycle that makes us incapable of electing anybody in the middle.