This is a common misconception. The referee has no control over who is in the stands - they are “outside agents”. Many people think that the referee can eject a spectator, but they can’t. The referee only has authority over the players and coaches.

This is why the first remedy is to ask the coaches to get their parents under control. In general, if the abuse continues, the coach can be carded as that’s the only person the ref can discipline.