They're not just average people, they're people with a particular condition which is more than likely associated with mental health issues, lack of social skills and several times more likely to be dealing with an addiction problem than a normal person.

Plus all the trust issues of having lived in the street. Only someone who hasn't interacted a lot with the homeless would say they are just like everyone else. Even if the reason they became homeless was just random by the time they've been homeless for a couple of years they are a different person.

There's a reason many of the homeless avoid shelters, if you talked to one you'd know why, and it's not because the other guests are lovely kind people to be around.

The bottom line is they have as much a right to be there as you do and you're free to ignore them or interact with them as much or as little as you want to.