The "never rose above a couple of thousand" small group refers to the number of activist Luddites. It doesn't refer to the people working in the textile industry in general - which was a big group, and which was heavily affected.