Being able to freely threaten reprisal against people exercising their rights circumvents those rights.

Freedom of association applies to individuals; it's a non-sequiter here.

It's relevant in that businesses generally also enjoy freedom of (non)association but obviously that's not an absolute.