I'd be annoyed and try to get them out of my space or leave myself. Is that bad?

Nope. That’s exactly what I’m advocating too.