Free movement is at the heart of the EU project. You start restricting that and the EU will tear up the agreements. We saw this already some 10 years ago when a similar vote passed, and EU stopped a lot of collaborations with the Swiss.