| ▲ | account42 2 days ago | |
Or optimize the website. A knowledge base that isn't being actively updated shouldn't need to hit PHP to serve unauthenticated requests. It doesn't excuse the scrapers but a lot of websites are just needlessly wasteful. | ||
| ▲ | edg5000 a day ago | parent [-] | |
From what I hear it's misconfigured bots that can cause issues. This could happen for dynamic content, where the bot thinks there are a lot of unique pages and it just keeps scraping with slightly different query params for example, sometimes not rate limited nor limited to pages/day/IP. That's my read from the various posts I've read from web admins. | ||