| ▲ | sippingabonedry an hour ago | |
How do you separate the Wayback Machine from malicious bots that pretend to be the Wayback Machine? Are you whitelisting their IP blocks? Because I get a ton of scraper requests that forge Googlebot, Bing, and Yandex user-agents that are totally not coming from their IP ranges. In fact, sometimes they all come from the same IP... | ||
| ▲ | Kodiack 11 minutes ago | parent | next [-] | |
They have their own ASN, which I’ve explicitly allowed requests from. | ||
| ▲ | fc417fc802 an hour ago | parent | prev [-] | |
I thought at least google (and possibly others) provided a way to verify the user agent? | ||