They are not contradictory. Publicly labeling someone a troll is indisputably speech. You mistake the situation as someone denying you your rights, but really you are just failing to recognize the same rights for that person or group.

The communities that are presumably excluding you are exercising their freedom of association.

You may not (indeed, seem not to) realize, but freedom of association is understood to be part of the set of freedoms generally known as freedom of speech/expression.

It's the freedom of people, collectively or individually, to associate with certain others, or not to associate with them.

You are reckoning with the consequences of how you've chosen to exercise your rights, but you have not been denied them. No one is obligated to include you (in the context of public discourse).