Well one does have to come up with continuous non-differentiable functions to begin with, right? Weierstrass had to shock the community with his weird series that's almost everywhere nondifferentiable before people could conceive of a nice framework that includes them. People do not invent whole encompassing abstractions out of nowhere

Great point, I think the argument you could make about Tao (fairly or unfairly) is he never tries to build that framework.