Maybe - a lot of the material wealth of the South was having a lot of land divided amongst fewer people. Enjoying more leisure has a nasty habit of not making people richer in the end.

Here's specifically what Adam Smith had to say in the Wealth of Nations:

> But if great improvements are seldom to be expected from great proprietors, they are least of all to be expected when they employ slaves for their workmen. The experience of all ages and nations, I believe, demonstrates that the work done by slaves, though it appears to cost only their maintenance, is in the end the dearest of any. A person who can acquire no property, can have no other interest but to eat as much, and to labour as little as possible. Whatever work he does beyond what is sufficient to purchase his own maintenance can be squeezed out of him by violence only, and not by any interest of his own.

Later, to explain this trap of why people insist on owning slaves even if paying workers would be more productive in the long run:

> "The pride of man makes him love to domineer, and nothing mortifies him so much as to be obliged to condescend to persuade his inferiors. Wherever the law allows it, and the nature of the work can afford it, therefore, he will generally prefer the service of slaves to that of freemen."