What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?

Edit: the article says millions of times per hour? (!?)

The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?

Feels like it would be a good time for freenet and the like to catch on.

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).

I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.

I think they also have to operate in countries that don't mind shady things like this.

TIL:

> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.

WTF. That's just botnets.

Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...

Thank god for residential proxies.

Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?

Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.

I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.

"He who has the gold makes the rules" is older than the pyramids.

> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.

We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.

https://news.ycombinator.com/newsguidelines.html

People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.