My impression was that there was a lack of fast moving rivers which were suitable for water wheels. You could make some elevation, or build a larger wheel, but that can become prohibitive for the volume needed for a real factory.

It looks like the south does have some suitable rivers, but you wonder why they exported their crops to the north just to buy them back again in their more processed form...that just doesn't make much sense from an economic standpoint. Clearly slavery wasn't a suitable replacement for the type of production work done in the north. It must have been a mix of social factors, combined with the fact that the north specialized in industry early on and you couldn't compete very well with the lack of expertise and lack of industry which supported the local industry in the south.

Anyway this is all just wild speculation. Take it for what you will.