Remix.run Logo
kfhfardin 7 hours ago

Hmm, I am sure you know more than I do on the topic. At least to me, an amateur in the security space, would the action of the scraper not be closer to repetitive calls to tables instead of, i.e a more inquisitive agent doing research or booking who spends more time looking through, and surely that can be tracked? Again, on the moral side, I agree that if one comes with the other, it is quite dystopian. I myself am an ML Inference Engineer, so I guess interesting problems and interesting solutions always draw me in much like this.

coldbrewed 6 hours ago | parent [-]

Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive.

At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.