Remix.run Logo
randunel 3 days ago

Why would scrapes get blocked, is scrapping illegal?

eitland 3 days ago | parent | next [-]

I don't know if it is, but I also don't think we are required to let dumb bots repeatedly assault or web sites if we can find a technical way to get around it.

Xylakant 3 days ago | parent | prev [-]

It's very often not, but it's still the website owners property and if they choose so, they can show misbehaving guests the door and kindly ask to remain on the other side (aka block them). Large scale scraping puts substantial burden on web properties. I was paged the other night because someone decided it would be a great idea to throw 200 000rq/s for a few minutes at some publicly available volunteer run service.