I grew up with servants (in Subsaharan Africa and Morocco).

However, they were paid (I have no idea whether it was a good wage, or not), and had pretty decent quarters (in Morocco). My parents were pretty kind, fairly liberal, people. I would be quite surprised (and shocked) if they took advantage of the servants. I know that my mother made damn sure that I had respect for poor folks.

Those things are a symptom of an undeveloped economy. It harkens back to a time of less development where there were more hands than jobs and much of the labor was manual. Not to excuse the practice in modern times but go back a few generations and that was the reality of the world -everywhere.