People are turning on immigrants because the pie has been shrinking for decades.