Remix.run Logo
nitwit005 4 hours ago

This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping.

The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.

ssl-3 3 hours ago | parent [-]

There are multiple goals at play. They're easy to find in HN comment sections whenever these topics arise.

One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all.

Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web.

These are not the same goals.