There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.

We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.

Most scrapers are relatively honest in some way shape or form.

▲

cogman10 an hour ago | parent | next [-]

One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]

[1] https://www.gentoo.org/news/2026/01/05/new-year.html

▲

sporp 41 minutes ago | parent | prev | next [-]

How did you implement this? My site's getting hammered, any tips would be appreciated

	▲	ComputerPerson 9 minutes ago \| parent [-]
		What kind of uptime/expectations/etc. are you working with? Scraper "attacks" don't take down our robot-specific server very often, so it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99.99% (that's a made-up statistic, but it's a very high number) of the time, the misdirected users don't realize anything is amiss. Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. FWIW I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our intended users. We have loads of other conditions, mostly banning specific IP ranges for entities when we know exactly who they are, but this is a good start.

▲

Symbiote an hour ago | parent | prev [-]

> Most scrapers are relatively honest in some way shape or form.

Did you miss a "dis" in there?