Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?

▲

simonw 8 hours ago | parent | next [-]

Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.

See https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.

▲

kees99 7 hours ago | parent | next [-]

This is exactly the problem, unfortunately.

For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].

Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.

[0] https://dumps.wikimedia.org/

	▲	nubinetwork 7 hours ago \| parent [-]
		You can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.

▲

perching_aix 7 hours ago | parent | prev [-]

Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.

▲

davidfischer 10 hours ago | parent | prev [-]

It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.

It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.

[1] https://docs.readthedocs.com/platform/stable/reference/markd...