Remix.run Logo
sporp 41 minutes ago

How did you implement this? My site's getting hammered, any tips would be appreciated

ComputerPerson 9 minutes ago | parent [-]

What kind of uptime/expectations/etc. are you working with?

Scraper "attacks" don't take down our robot-specific server very often, so it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99.99% (that's a made-up statistic, but it's a very high number) of the time, the misdirected users don't realize anything is amiss.

Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. FWIW I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our intended users.

We have loads of other conditions, mostly banning specific IP ranges for entities when we know exactly who they are, but this is a good start.