| ▲ | semiquaver 5 hours ago |
| > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds. Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task: > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
|
|
| ▲ | xena 6 minutes ago | parent | next [-] |
| The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng... |
|
| ▲ | apitman 18 minutes ago | parent | prev | next [-] |
| This comment captures the current AI zeitgeist about as well as anything I've seen. |
|
| ▲ | wongarsu 4 hours ago | parent | prev | next [-] |
| Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this |
| |
| ▲ | semiquaver 4 hours ago | parent | next [-] | | > I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders? | | |
| ▲ | paytonjjones 4 hours ago | parent [-] | | In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course. | | |
| ▲ | pbronez 3 hours ago | parent [-] | | It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses. | | |
| ▲ | paytonjjones 3 hours ago | parent [-] | | It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path. | | |
| ▲ | inigyou 2 hours ago | parent [-] | | Why aren't we litigating against scrapers, anyway? DDoS is a felony. | | |
| ▲ | ekidd an hour ago | parent [-] | | Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners. | | |
| ▲ | inigyou an hour ago | parent [-] | | If you block Brazil, they'll find an alternative, maybe then you can sue them. |
|
|
|
|
|
| |
| ▲ | Aurornis 3 hours ago | parent | prev [-] | | > Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time. So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views. A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost. The amortized per-view cost is highest for the real user. | | |
| ▲ | rplnt an hour ago | parent [-] | | Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished. | | |
|
|
|
| ▲ | DenisM an hour ago | parent | prev | next [-] |
| Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once. Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset. |
| |
|
| ▲ | inventor7777 2 hours ago | parent | prev | next [-] |
| My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining? |
|
| ▲ | karlgkk 2 hours ago | parent | prev | next [-] |
| > I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. You yourself describe the solution. Extensions or browser integrations of the Anubis task. |
| |
| ▲ | dpark an hour ago | parent [-] | | He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster. |
|
|
| ▲ | Self-Perfection 4 hours ago | parent | prev | next [-] |
| The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS? But it would just make Anubis suck a little bit less Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low. |
| |
| ▲ | bayindirh 2 hours ago | parent [-] | | > I suppose it does not even have ability to bypass hashing screen when server load is low. Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases. |
|
|
| ▲ | radlad 4 hours ago | parent | prev | next [-] |
| I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot. |
| |
| ▲ | jwilk 3 hours ago | parent [-] | | Huh? Are you saying there's an Argon2 implementation in the kernel? | | |
|
|
| ▲ | smallerize 5 hours ago | parent | prev | next [-] |
| But the scraper is making way more requests and is paying for all that compute. |
| |
| ▲ | tptacek 4 hours ago | parent | next [-] | | This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers. That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards". | | |
| ▲ | progval 3 hours ago | parent | next [-] | | > There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers. There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over. | | | |
| ▲ | graemep 4 hours ago | parent | prev | next [-] | | I wonder whether what Anubis is doing might be blocking agents that do not run JS or that have a timeout (or humans who give up as I just did) | | |
| ▲ | tptacek 4 hours ago | parent [-] | | I suspect that's most of the value, and that the deterrent is about software construction and not about the proof-of-work, which I've been saying for the last year is a productive direction for Anubis to head towards. Instead, it's just sort of tormenting normal people, as a kind of performance of anger about AI. Which: I get! If it's your site, perform all the anger you want. |
| |
| ▲ | cyberax 2 hours ago | parent | prev [-] | | I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute. Anubis doesn't really help against this. |
| |
| ▲ | semiquaver 4 hours ago | parent | prev | next [-] | | no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week. | | |
| ▲ | NobodyNada 4 hours ago | parent [-] | | The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs. | | |
| ▲ | semiquaver 4 hours ago | parent [-] | | OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site. a $5000 ASIC-based device can hash SHA256 at 200TH/s or more. | | |
| ▲ | inigyou 3 hours ago | parent | next [-] | | The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP. | |
| ▲ | dspillett 2 hours ago | parent | prev | next [-] | | > In other words not a meaningful barrier. And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target. > a $5000 ASIC-based device can hash SHA256 at 200TH/s or more Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves. | |
| ▲ | LatticeAnimal 4 hours ago | parent | prev [-] | | Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle... | | |
| ▲ | inigyou 3 hours ago | parent | next [-] | | You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses. | | | |
| ▲ | CursedSilicon 2 hours ago | parent | prev [-] | | ...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of? |
|
|
|
| |
| ▲ | DarmokTanagra 3 hours ago | parent | prev [-] | | not if its running on someone else's device |
|
|
| ▲ | rzmmm 4 hours ago | parent | prev | next [-] |
| It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency. |
| |
| ▲ | inigyou 3 hours ago | parent [-] | | And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge. | | |
| ▲ | wbl 2 hours ago | parent [-] | | Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones. | | |
| ▲ | applfanboysbgon an hour ago | parent [-] | | Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with. | | |
| ▲ | Terr_ 35 minutes ago | parent [-] | | Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together. Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else. |
|
|
|
|
|
| ▲ | TZubiri 4 hours ago | parent | prev [-] |
| I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental. When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it. |
| |
| ▲ | lxgr 4 hours ago | parent [-] | | I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero". Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead). | | |
| ▲ | TZubiri an hour ago | parent [-] | | I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed. |
|
|
