Plenty of us remember when Switzerland was not in the Schengen though. This might be good for Americans, who haven't been able to get working visas in Switzerland since EU countries now have priority. But otherwise, I don't see much changing beyond border procedures.

As with Brexit, leaving is likely to result in a much stricter regime than the status quo from before the establishment of the system.

Most of the downsides would be on the goods side. Swiss companies would loose market access and the chance of "better" trade agreements is even worse then the UK, especially currently.