Remix.run Logo
mitxela 2 hours ago

Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.

XorNot 2 hours ago | parent [-]

Scrapers scrape every link they can find.

The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.

So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.

mitxela 2 hours ago | parent [-]

The main reason to have those links is for people to click on them. No sensibly programmed machine is scraping diffs from a git web front end.