> I'm really not sure the "legion of servants" of Victorian-era elites were the people experiencing the worst living conditions on earth.

Indeed not. Being in service provided one of the most desirable careers you could hope for as a working class Victorian. When your options look like pit, mill, or factory, cleaning floors in a nice warm house is a good alternative. And that was just the foot in the door: there was progression available. Those jobs were hard to get, prestigious, came with their own status hierarchy, and could keep you set for life.