You will encounter homeless people in libraries, because it's one of the few public spaces that won't kick them out. Your reaction to that shouldn't be to hate and avoid libraries though. It should be to appreciate them more.

The guy didn't said he hated it, you did. He just said he avoided it. I would too. The same way I wouldn't want to hang out at a homeless shelter (and why many homeless themselves avoid any places with many other homeless people).