That depends on the definition of "surveillance". Should a foreman not pay close attention to his workers? Should a hospital not track its patients' locations and vital stats while within the hospital? Are cameras in a jewelry shop morally wrong?

Your neighbor's surveillance of you is bad because they're violating your privacy, and using the tool of surveillance to do it. If you lived in a foggy area and they were monitoring their front walkway with a camera that was good at seeing through fog, and they happened to get a corner of your property in the camera's field of view, then you might have something to complain about but I wouldn't call it morally wrong.

I agree that surveillance is a tool of control. So are fences. It's ok to control some things.

I also agree that surveillance gets into sticky territory very, very quickly. I definitely don't have a clean dividing line between what I'd like the police to be able to see and what they shouldn't. (Especially when the temptation to share that data is so strong and frequently succumbed to.) I would probably say in some useless abstract sense, mass surveillance is also morally neutral. But given that it's proven to be pretty much impossible to implement in a way that doesn't end up serving more evil than good, I wouldn't object to calling it immoral.

IMHO surveillance is a problem when it is asymmetric ; which is obviously the case here. Governments for example are watching everyone inside and outside, but the people that are being monitored simply cannot really watch the people watching them. Don't you agree ?

In this view, maybe an ultra radical solution to privacy issues is : no privacy at all, for no-one. Complete and total transparency of everyone to everyone. Now the question is how to implement that ? That's obviously impossible, because someone in power will always have something to hide. So maybe if true democracy where everyone holds exactly the same amount of power that could work ? Same issue, because it is impossible to implement too. Oh well.

if its ok for the foreman to control the workers, you would then say its ok for the foreman to hold the workers at gun point while they work?

id say the control is immoral, in all forms. Voluntary agreement and consent are fine but then its not surveillance, its people saying where they are. the patient wants the doctor to know where they are and what they are doing, and not just letting the doctor decide on their own what to know.

the worker wants the foreman to know that they are present and working, in fulfilment of their contract together. its not surveillance either.

the jewelry store itself is immoral, but private property and control thereof is a tradeoff we've made

Again, there are plenty of instances where enough good comes from surveillance that it outweighs the intrinsic negative, but denying that it is, in of itself, intrinsically negative suggests that some creepy dude monitoring everyone's every move is just fine, as long as he's not doing anything else.

A more obvious parallel is violence. To trip over Godwin's law, shooting Hitler would have been a moral action, but not because "shooting people" is amoral. Shooting a random person is definitely immoral. We constantly do immoral things for the greater good, but it is a mistake to thusly assume those actions are amoral.

> So are fences

Good fences make good neighbors... If I could put a notion in his head: Why do they make good neighbors?