I doubt many refs want this responsibility, even if it were feasible. And I don't think it is feasible in most cases. For instance, how is the ref going to pick out the problem fans while also managing the game itself? And how is the ref supposed to enforce the removal?

Keep in mind these refs are often volunteers, working solo, have no official connection to the teams or field owners.

And there is a ref shortage virtually everywhere, at all levels and sports. Making it more difficult just isn't realistic.