Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...

▲

mattmcal 3 hours ago | parent [-]

There is a section in the article answering your question if you read it.

> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.

▲

mbirth 3 hours ago | parent | next [-]

I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.

	▲	inigyou an hour ago \| parent [-]
		abuseipdb.com Some ISPs ban customers based on a single report there - have fun!

▲

fer an hour ago | parent | prev [-]

If you operate your site alone in a silo yes, but crowdsourced solutions to ban/rate limit those IPs exist.