High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?

> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

▲

KomoD 4 hours ago | parent | next [-]

> High Anubis difficulty is annoying the hell out of me for several sites.

I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.

	▲	johneth 4 hours ago \| parent [-]
		Just look at another tab while you're waiting if you're that bothered.

▲

sethops1 6 hours ago | parent | prev | next [-]

In a few years the VC money will dry up and this gross overspend on slurping data will end.

▲

igor47 5 hours ago | parent | next [-]

Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls

▲

marginalia_nu 4 hours ago | parent | prev | next [-]

It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.

	▲	inigyou an hour ago \| parent [-]
		And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare. I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.

▲

pixl97 4 hours ago | parent | prev [-]

Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.

▲

MithrilTuxedo 5 hours ago | parent | prev [-]

Has anyone considered having Anubis perform more valuable hashing?

Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.

▲

IshKebab 5 hours ago | parent [-]

Yeah that has been around for many years. Usually by sketchy download sites.

It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.

	▲	inigyou an hour ago \| parent [-]
		Yes they are. The proxy is just a proxy. All processing is central.