Remix.run Logo
kro 4 hours ago

It does not even require the PoW thing Anubis does. I've setup a simple logic that just:

Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.

Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op

I've extended this with a FCRDNS checked exclusion for Googlebot.

Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.

Also, I only apply these rules to routes that are not cheap and cached.

harshreality 3 hours ago | parent [-]

That's not far from what anubis does for clients that are determined to have light souls. It doesn't always send a PoW challenge.

For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.