Remix.run Logo
aeturnum 4 hours ago

Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.

nickgray 4 hours ago | parent [-]

OP here: I'm not scraping thousands of times per day! Usually just a few times per year.

jader201 2 hours ago | parent [-]

But you could just be one of thousands of bots targeting the same sources you’re scraping.