In my experience, the phrase is just used to mean, "I don't hate immigrants, but..." (which, like the phrase "I'm not racist, but...", you are free to doubt case-by-case). I.e. it is not inherently inconsistent to apply the same disclaimer regarding a belief that legal immigration is too loose, too high, mismanaged, whatever; since that doesn't necessitate a belief that immigration as a concept is bad.