I understand. However, I interpret your statement as coming from your position: as a moderator, with a job to do as a curator of a community.
How does the community learn what to flag?
It's not about piling on. You can fix disruptive behaviour by picking off remarkably few individuals, but that behaviour is easy to classify. Everybody knows what a troll looks like. But you can't teach a community what is generally acceptable and unacceptable within the culture of the community without a conversation in public.
Consider two forms of justice for a new rule: one day, it becomes illegal to smoke in restaurants.
In the first form of justice, in the interest of avoiding pile-ons and vigilantism, secret police turn up at your home in the evening and confiscate your cigarettes, or fire you from your job, or simply disappear you. The right way to deal with smoking isn't to say anything in public; it's to inform your friendly secret police and they'll deal with it. They'll look for a pattern and they'll decide the punishment. The offender gets fixed. The decision isn't public, so it can't be criticized either. If a few visiting foreigners also get abducted because they didn't know what the norm was, well, tough.
In the other form, justice must not only be done, but it must be seen to be done. The conversation happens in public. Everyone is empowered to teach everyone else about norms. A friendly word to the folks from out of town let's them know what's what. The community finds its own level of comfort for where the boundaries are. Not everyone is happy, but everybody can see what's going on.
Now I'm not saying you're like secret police. You do however have a similar working model. You're not accountable to the community and you want to make decisions in private without any second-guessing. You mean well. But your tools are more suited to dealing with trolls than finding and socially enforcing community norms. I don't envy you. I would not like to be a moderator.
Best of luck.