I work with people who actively chose to go to conference in Berlin instead of the UK due to visa issues. The passport control is a real issue one cannot just handwave away.