I've seen websites put up some draconian measures to try and get a grip on the scraping. So much for the sub-second loading experience when you have Cloudflare, Google, Anubis, and all these other captcha services trying to see if you're a human. It's made the web browsing experience so much worse.

Some of the proposals to address this include charging bots for access to web resources, but they will also have repercussions for regular users. I don't see how you solve this cleanly.

▲

robinsonb5 2 hours ago | parent | next [-]

Sure - it sucks, unfortunately the alternative is the sites going away entirely. When the load from scraper bots is constantly knocking the site offline the choices are literally to allow it to remain inaccessible for much of the time, put up a layer of defenses with all the user-annoyance compromises that entails, or just give up and unpublish the site.

▲

Borg3 2 hours ago | parent [-]

The alternative is simple.. Go dark. VPN tech is known from like 30 years. Pretty much everyone can use it (VPN providers). But instead using it to browse net, build VPN overlay networks of interest for people. Gaming networks, R&D networks, Retro Networks. People will peer to PoP and use resources. Bad actor? BAN it from network. You have control. This could be done in Internet, but big corpos and big money won the battle. Just wake F*ing up...

	▲	kukkeliskuu 22 minutes ago \| parent [-]
		Continuing on your suggestion. There could be open source tooling to create custom private "closednets", with - trust ring mechanism to allow invitations, flagging, banning, and banning those that invite people who were banned - the rules of the closednet - search engine with opt-in scraping - portal (remember the 80s?) with all the registered nodes, perhaps by service category such as public git repo hosts, web sites etc. etc. The first closednet could be Hacker News.

▲

Grimburger 2 hours ago | parent | prev | next [-]

Cloudflare specifically has a block for LLM and AI training bots now.

Not sure of the effectiveness but it's there.

	▲	matherial 2 hours ago \| parent \| next [-]
		Minimal. I'm behind Cloudflare and 90% of the traffic is still scrapers. I don't think they're serious about the long tail. I think the main thing Cloudflare is trying to do is block direct traffic from frontier labs and then start charging them for access. They might end up shooting themselves in the foot, as this simply empowers sketchy residential-proxy outfits to undercut Cloudflare and sell the data to labs for less.
	▲	bakugo an hour ago \| parent \| prev [-]
		It still only blocks "well-behaved" bots that have proper User-Agents and respect robots.txt, so it's largely pointless. The problematic bots are all disguising themselves as Chrome and sending requests from millions of residential proxy IPs, and the only real solution to those is some sort of captcha or PoW page on first visit.

▲

JKCalhoun 8 hours ago | parent | prev [-]

Me, I'm just scraping the parts of the internet I like, toying with local LLMs… ready really to just shove off.