Remix.run Logo
account42 2 days ago

Or optimize the website. A knowledge base that isn't being actively updated shouldn't need to hit PHP to serve unauthenticated requests.

It doesn't excuse the scrapers but a lot of websites are just needlessly wasteful.

edg5000 a day ago | parent [-]

From what I hear it's misconfigured bots that can cause issues. This could happen for dynamic content, where the bot thinks there are a lot of unique pages and it just keeps scraping with slightly different query params for example, sometimes not rate limited nor limited to pages/day/IP. That's my read from the various posts I've read from web admins.