I disagree. The kernel finds it effective - 66% of scrapers are turned away directly.

The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.

If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.

▲ tptacek 7 hours ago | parent | next [-]

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

	▲	yorwba 3 hours ago \| parent \| next [-]
		For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
	▲	6 hours ago \| parent \| prev [-]
		[deleted]

▲ Y_Y 7 hours ago | parent | prev | next [-]

The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.

▲

wongarsu 5 hours ago | parent [-]

That's still a notable step up in completely and resource investment for the crawler

See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha

	▲	inigyou 3 hours ago \| parent [-]
		That's not a step up, that is how they work by default.

▲ oasisbob 5 hours ago | parent | prev | next [-]

> Many of these boxes may not even have the spare CPU power ...

I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.

Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.

If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.

▲ lxgr 6 hours ago | parent | prev | next [-]

Why would they even run a browser engine on the devices they're hosted on?

All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.

▲ lxgr 6 hours ago | parent | prev | next [-]

> 66% of scrapers are turned away directly.

Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.

The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.

[1] https://news.ycombinator.com/item?id=49500040

▲ semiquaver 5 hours ago | parent | prev | next [-]

  > meaningful dent

Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.

▲ graemep 6 hours ago | parent | prev [-]

How s that measured? How do you count human users who have been turned away?