This is fair, as long as you're a small operation, and those you didn't like can go shop elsewhere. Visa + MasterCard are a duopoly controlling the overwhelming majority of the market, and if they ban a seller, there's little else the seller can switch to.

I think a processor like Visa could benefit a lot from the status similar to the "common carrier". Like a telephone network must offer service to anyone, but cannot be held liable for the content of the communication (even if criminals are using it), Visa could accept the requirement to pass payments to any counterparty in exchange for dropping the KYC requirements. Let banks and merchants care about that.

I don't think that the US or EU government would agree to grant such a status though.