The problem with small refactors over time is that your information about what constitutes a good/complete model of your system increases over time as you understand customers and encounter edge cases. Small refactors over time can cause architectural churn and bad abstractions. Additionally, if you ever want to do a programmatic rewrite of code, with a bunch of small refactors that becomes more difficult, with a single surface you can sometimes just use a macro to change everything all at once.

This is an example of a premature optimization. The reason it can still be good is that large refactors are an art that most people haven't suffered enough to master. There are patterns to make it tractable, but it's riskier and engineers often aren't personally invested in their codebases enough to bother over just fixing the few things that personally drive them nuts.