Judges who violate the basic rights of other judges would also be subject to some level of accountability though. At a certain point, we have to trust government officials to at least attempt to do their jobs and we need to have ways to address the situation when they don't. It shouldn't matter if that judge is a democrat or a republican.
Right now there is currently zero accountability. At best, when a judge violates people's constitutional rights some small number of those people will be able to get an unjust ruling overturned at which point they might be released from prison or might get some monetary payout at the expense of taxpayers, but the judge is still free to do whatever they want without consequence knowing that at least a few people will be unable to assert their rights.
Considering that unaccountable judges are where we're starting from, I think having a means to make judges accountable can only improve things. Given the choice between judges being able to violate people's rights without any accountability or a system where judges have some level of accountability for the most egregious violations of our rights, even while that system requires us to make sure that it isn't being clearly abused, I think we're better off with the option to get some accountability where it's needed.
It doesn't need to be a perfect system to be a better one, and it feels like we could put some guardrails in place to keep the amount of obvious abuse down. It's difficult to believe that judges willfully violating people's rights without consequence is an unsolvable problem, let alone one that couldn't possibly be improved somehow.