Broadly, I agree with your sentiment. As soon as some people rule over others, given enough time, things creep towards total enslavement and disenfranchisement of the others. This has been proven over and over.

The question then becomes, how do we organise society instead?