Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.

See https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.

This is exactly the problem, unfortunately.

For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].

Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.

[0] https://dumps.wikimedia.org/

Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.