Wouldn't that drive textile manufacturing out of the EU?

I think that's the point. EU doesn't actually want these factories to operate in the EU. They just want to buy the clothes from elsewhere.

But then of course they cry a lot when they realize how easy it has become for China and USA to squeeze them.

Tough consequences of stuff like this.

No, this will give EU textile manufacturing an advantage, because they mostly produce low volume high value products. This will hurt fast fashion importers the most.