You could also publish a list of IP addresses.

As the article describes, it doesn't help, because the traffic originates from millions of unique residential IPs across hundreds of ASNs and countries.

▲

inigyou 3 hours ago | parent [-]

So?

	▲	VladVladikoff an hour ago \| parent [-]
		Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma