Remix.run Logo
tadfisher a day ago

Does there need to be a principled justification beyond that? I used to be on the side of the traffic, as in, it does not matter where traffic originates as long as it's not abusive. But the fact is that too many scrapers exist which are, in fact, bad. Their behavior is bad, their programming is bad, and they result in way too high costs for free infrastructure, thus they are morally bad.

I expect AT&T and Comcast to offer a residential proxy service any day now.

topranks 20 hours ago | parent [-]

Absolutely.

Bear in mind the scrapers wouldn’t need to use these proxies were they not being blocked by the sites they are scraping. So it’s being used to evade blocks.

For some content the level of scraping is outweighing real users, driving up costs and pushing them towards more closed models.

Wikipedia for example make content available free, if you start hammering the site they will rate limit you to keep the lights on. If you need the data fast in bulk they have a paid program to get it without scraping. But some prefer to neither adhere to reasonable request limits nor pay for their use of the infra; instead they choose to pay these grifters to avoid the rate limits.