> Why is git.kernel.org “interesting” to crawlers

Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.

wiredfool 5 hours ago | parent | next [-]

Seeing the exact same thing on (somewhat high profile) open data sites I run.

The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.

Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.

▲

inigyou an hour ago | parent | next [-]

Once detected, don't block them because they'll just change strategies automatically, but you can toy with them, like returning a page full of random numbers instead of real data.

	▲	wiredfool 9 minutes ago \| parent [-]
		I’d love to, but I’m a bit limited in what I can do from a reputational damage POV. They’re my sites, in that I’m responsible, but they aren’t something where I can return incorrect responses.

▲

marginalia_nu 4 hours ago | parent | prev [-]

Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.

▲

inigyou an hour ago | parent [-]

That would be a different bot, one written specifically for your site. Mainly we're discussing the dumb ones that just crawl all possible http links

	▲	marginalia_nu an hour ago \| parent [-]
		No I think this is the same case. It seems to just be following local links on the SERP. From a search for jack parsons, you can find hyperlinks to the sorts of requests it's making.

▲

iririririr 5 hours ago | parent | prev [-]

so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.