| ▲ | topranks 20 hours ago | |
Absolutely. Bear in mind the scrapers wouldn’t need to use these proxies were they not being blocked by the sites they are scraping. So it’s being used to evade blocks. For some content the level of scraping is outweighing real users, driving up costs and pushing them towards more closed models. Wikipedia for example make content available free, if you start hammering the site they will rate limit you to keep the lights on. If you need the data fast in bulk they have a paid program to get it without scraping. But some prefer to neither adhere to reasonable request limits nor pay for their use of the infra; instead they choose to pay these grifters to avoid the rate limits. | ||