I run some small websites, including a tiny forum that’s been a goldmine for scrapers. I had to significantly tweak some firewall rules and configuration after scrapers behind residential proxies suddenly accounted for over 99% of requests.

However, I also relaxed rules for automated traffic that was well-behaved, and I went out of my way to ensure that the Wayback Machine was able to hit everything. I should kick a small donation their way. They provide an incredibly valuable service and I love the benefit that I get from them just for personal side projects.

▲

sippingabonedry an hour ago | parent | next [-]

How do you separate the Wayback Machine from malicious bots that pretend to be the Wayback Machine? Are you whitelisting their IP blocks?

Because I get a ton of scraper requests that forge Googlebot, Bing, and Yandex user-agents that are totally not coming from their IP ranges. In fact, sometimes they all come from the same IP...

	▲	Kodiack 11 minutes ago \| parent \| next [-]
		They have their own ASN, which I’ve explicitly allowed requests from. https://www.peeringdb.com/asn/7941
	▲	fc417fc802 an hour ago \| parent \| prev [-]
		I thought at least google (and possibly others) provided a way to verify the user agent?

▲

mkatx 2 hours ago | parent | prev [-]

This is the way to go! Cut the cat and mouse, win win ish.