I've been posting about this recently but I continue to be bad at either timing my submissions or titling them. I struggle with YouTube thumbnails too so I think it might be me. But anyway:

I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.

I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.

▲

bayganyo 2 hours ago | parent | next [-]

I like the spirit of this approach; make them suffer! But I'm surprised the bots wouldn't have reasonable timeouts to counteract. How long do they typically wait on these requests?

EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!

	▲	robotmay an hour ago \| parent [-]
		I should probably put together some actual stats at some point, but from what I see in the live logs you'd be shocked - some of them will happily sit there for over 5 minutes being drip-fed packets! Looking at you, Meta. I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning). I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.

▲

pmlnr an hour ago | parent | prev [-]

> 3000 concurrent crawlers

I gave up on tarpits when it got to 130k "waiting" connections.