Remix.run Logo
▲ dspillett a day ago

If you are rolling your own it probably doesn't need to be as clever as Anubis is trying to be. Simple something relatively CPU heavy in JS will do the trick, bonus points if you can make it memory-heavy too. Most bots will give up running something CPU heavy currently (though they will run a lot of JS stuff as much of the web is inaccessible without) and they aren't going to optimise your challenge in WASM or similar because it isn't worth it for the small number of sites it protects (Anubis' challenges on the other hand may well be worth optimising for that way because of their wider use, hence that project is stuck in the cleverness war of attrition and definitely needs to move toward memory-hard problems and something more efficient than JS sooner rather than later).

My main reason for planning to implement something like that myself instead of something off the shelf is two-fold:

1. The points mentioned above

2. I can be a dick and make some subsets do harder work (longer challenges, adding time delays that wouldn't phase a bot) - “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”

▲8by3 16 hours ago | parent | next [-]

The challenge isn't terribly clever, just some basic crypto maths, same as Anubis. There are a very low number of AI bots that pass it based on my logs, like 0.01% low and I suspect they are interactive agents that bother to run JS as crawlers typically don't. Assuming these are actually run on an end users computers I don't think any amount of cpu/memory weight would be a real blocker. One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.

▲dspillett 14 hours ago | parent [-]

> that bother to run JS as crawlers typically don't

I'd be surprised if many don't run basic JS, given how many sites require it for basic content these days (load base site, then make a fetch request for the actual content), unless there are less such sites than I think.

> One thing I've seen that is interesting'ish is bots failing as they answer the challenge from a different IP than it was requested from, and the source IP is validated as part of the response so they fail.

From what little I've looked into the details, it seems that scrapers (both old and new) using a scattering arrangement (built in or via an outgoing proxy) seem to be split 75/25 between those that completely scatter (following requests will almost certainly come from a different address) and those that stay on the same address for at least the first few requests (which would cover login, bot challenge, or such) then move around to avoid requests-per-source-address based resource management measures. Only really simple ones (someone running a recursive wget without proxy) stick to the same source address all the time now.

Unfortunately you can't be more strict about changing source addresses because of how many false positives you will see. CGNAT arrangements often move people between addresses fairly regularly, and even without that there is the common case of people moving away from a WiFi AP and automatically flicking to their mobile network (or another nearby AP).

▲jacobgkau a day ago | parent | prev [-]

> “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”

I'm assuming you're outside the US based on having this idea.

You might as well just say "I'm mad at your country for the tariffs, so here's an extra wait." From the fact you feel the urge to spite people in a tariffing country like this, it's clear the tariffs have had some negative effect on you, which goes against your narrative that they're "effectively paid for" by those in the tariffing country who voted for it.

▲dspillett a day ago | parent [-]

You have a some incorrect assumptions there. Yes, I'm outside the US, though I can't say I've experienced any issues at all that can be attributed to the continuing little tariff spat. I have heard of US citizens either complaining about cost raising because of them, or if not complaining then acknowledging them as something they consider a necessary pain. There are a fair few things I'm mad about, some of which do stem from the current administration over there, but this certainly isn't one of them. In fact I'm currently amused by it and the urge to mildly irritate those state side is in order to extend that amusement a touch.

▲jacobgkau 7 hours ago | parent [-]

I see, you're supposedly just amused by annoying people... that certainly makes you look better, I can see now why I was downvoted. /s

If you think your nagware is going to get back at US citizens for completely unrelated policies you're mad about, then maybe you should list those policies in your message instead of bringing up tariffs? That seems like it would be more productive to your goals, and less likely to make it look like you're salty and/or making a faulty argument regarding tariffs.