Sure, you don't need to "believe" the same things they do, but if you are vocal about your ideals on Twitter, for instance, you are unwelcome.

"Don't publicly disagree with my ideals or I will use my little fiefdom to destroy you."

You might be letting your personal experiences pollute things. The CoC never says that. It's anti-harassment language, and specifically enumerates what that might look like. Some contributors are members of the LGBT community, and I'm sure don't want to debate their personal identity when they're there to contribute to a project and write code.