Remix.run Logo
▲ hephaes7us 15 hours ago

Presumably, if you pay and respect rate limits, you get to bypass the anti-scraping measures?

Admittedly, the price that most will be willing to pay is indeed capped by the cost of the alternatives (bypassing anti-scraping measures via proxies, etc). Still, that means that there's a nonzero rational price.

▲cvadict 14 hours ago | parent | next [-]

> Presumably, if you pay and respect rate limits, you get to bypass the anti-scraping measures?

Far cheaper to just use your hyper-intelligence machines to bypass the anti-scraping measures than to pay everyone.

As of today you can only block the scattershot scrapers trying to slurp up everything at minimal effort/cost. Thankfully those make up the vast majority of traffic, so even dead-simple measures like Anubis block them. Whereas an advanced scraper is getting through any technical measures you can conceivably put in place (including the SOTA paid/cloud offerings). At the high end, the agents just drive regular browsers, mobile device, etc. running through a residential proxy / mobile CGNAT'd ip, solving captcha's in real-time, reasoning about the protection measures in place + adapting in real-time, etc. etc. They are virtually indistinguishable from humans. If you want humans to be able to see your content, the bots will too, as your average bot can now be far smarter + faster + adaptable than your average human.

▲warkdarrior 13 hours ago | parent | prev [-]

> Presumably, if you pay and respect rate limits, you get to bypass the anti-scraping measures?

Payment are not made for "presumably". The website should offer a clear contract: "to turn off anti-scraping measures for X amount of time, pay $Y".