Remix.run Logo
edg5000 2 days ago

They're not yet gone based on the message. They just have to adjust their hardening. The title is a bit dramatic.

account42 2 days ago | parent [-]

Or optimize the website. A knowledge base that isn't being actively updated shouldn't need to hit PHP to serve unauthenticated requests.

It doesn't excuse the scrapers but a lot of websites are just needlessly wasteful.

edg5000 a day ago | parent [-]

From what I hear it's misconfigured bots that can cause issues. This could happen for dynamic content, where the bot thinks there are a lot of unique pages and it just keeps scraping with slightly different query params for example, sometimes not rate limited nor limited to pages/day/IP. That's my read from the various posts I've read from web admins.