Your post-rationalization just doubles down on the stance that these crawlers are abusive and poorly developed.

You're also under the blatantly wrong misconception that people are worried about their data, when they are worried about the load of a poorly configured crawler.

The crawler will scrape the whole website on a regular interval anyway, so what is the point of this "optimization" that optimizes for highly infrequent events?