Yeah at that point you are almost obligated to confront the person who was trying to avoid conflict. Like, dude, just say something. Assuming the worst is a very bad habit and very antisocial.

It should be noted that trespassing near a residence at night, is often different than during the day. Where I live, at night it's criminal.