How having lot of sex would cause harm to my neighbours?

They might get sad because they feel like they're missing out

Puritanism: The haunting fear that someone, somewhere, may be having a good time.

I'm sure OP would argue that the breakdown of the family structure and social cohesion has a deleterious effect on society as a whole, which includes your neighbors.