From my experience, the refs typically assign responsibility to the head coach of the team the fans represent. It's not the refs job to manage the fans. They are there to manage the game. If the coach or staff determine the problem fans should be removed, they can do that.