In my experience, Swiss don't like criminals, unemployed people and people showing openly their religion. They negatively associate certain nationalities with stereotypes (e.g. Albanians, Maroccans etc.). If you are a representative of these groups, yes, it will be a problem. Violence towards foreigners is, compared to other countries, does not exist. Also with other nationalities, it is very different. Some people don't like Germans (that's also historically of course). However, with Germans near the boder it is often not a problem because they are more similar (and know how Swiss behave). With people from Berlin, many Swiss have not much in common. My wife is visible not Swiss and she never encountered raciscm (quite the contrary, she gets more free products at local stores than me because people recognize her). She also likes to buy tomatoes only from Switzerland. It is all how you behave in my experience. To the SVP, it is quite a different between the party and representative that are in the government. They are considered moderate due to the political system in Switzerland.

The SVP is not considered "moderate". They are a far-right party. The fact that they are wide spread and gather a lot of votes does not make them "moderate".

Source: am Swiss

As an anecdote re Germans: A friend of mine did an Auslandssemester there and was surprised to see "No Germans" signs for some of the housing options. Always makes me chuckle as an example how "relative" xenophobia is.