I would avoid homeless shelters because I have no reason to be there. But I won't avoid libraries because I do have a reason to be there, and I know there's no reason to be scared of interacting with homeless people.

They're just people and the library is for them too.

They're not just average people, they're people with a particular condition which is more than likely associated with mental health issues, lack of social skills and several times more likely to be dealing with an addiction problem than a normal person.

Plus all the trust issues of having lived in the street. Only someone who hasn't interacted a lot with the homeless would say they are just like everyone else. Even if the reason they became homeless was just random by the time they've been homeless for a couple of years they are a different person.

There's a reason many of the homeless avoid shelters, if you talked to one you'd know why, and it's not because the other guests are lovely kind people to be around.