if somebody else doesn't want to be part of your little club, why should you get to take their house away for that?

Because they signed the paperwork saying they wanted to be part of the little club and gave that little club the power to take away their house.