But I think the key difference is the potential for harm and power imbalance involved

You're talking about the consequences of consent being violated. I'm asking something different. If "the reason why a drunk person can't consent to sex is because their judgement is impaired", then there's nothing in that sentence that makes sex special. Replace it with anything else and it's equally true. "The reason why a drunk person can't consent to a loan is because their judgement is impaired." That the consequences of consent being violated in one instance or another are different doesn't change the fact that consent to anything has been defined as impossible in that situation.