There's nothing wrong with technical debt per se. As with all debt, the problem is incurring it without a plan or means to pay it off. Debt based financing is the engine of modern capitalism.

Gradual growth to large scale implies an ongoing refactoring cost--that's the price of paying off the technical debt that got you started and built initial success in small scale rollouts. As long as you keep "servicing" your debt (which can include throwing away an earlier chunk and building a more scalable replacement with the lessons learned), you're doing fine.

The magic words here to management/product owners is "we built it that way the first time because it got us running quickly and taught us what we need to know to build the scalable version. If we'd tried to go for the scalable version first, we wouldn't have known foo, bar and baz, and we'd have failed and wouldn't have learned anything."