I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).

Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.

▲

nxndbebdb 6 hours ago | parent [-]

Almost all of my visits to cgit instances are through direct deep links. Hard to imagine someone randomly browsing git listings

	▲	nneonneo 6 hours ago \| parent [-]
		The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links. Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.