The Victorians were talking about “ladies”, not the washerwomen and cooks. Ladies are delicate and slight.
The earthy workers existed to toil, not be beautiful. That wasn’t their station in life.