Remix.run Logo
coldbrewed 8 hours ago

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.

kfhfardin 7 hours ago | parent | next [-]

Hmm, I am sure you know more than I do on the topic. At least to me, an amateur in the security space, would the action of the scraper not be closer to repetitive calls to tables instead of, i.e a more inquisitive agent doing research or booking who spends more time looking through, and surely that can be tracked? Again, on the moral side, I agree that if one comes with the other, it is quite dystopian. I myself am an ML Inference Engineer, so I guess interesting problems and interesting solutions always draw me in much like this.

coldbrewed 6 hours ago | parent [-]

Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive.

At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.

deckar01 7 hours ago | parent | prev [-]

It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.

coldbrewed 6 hours ago | parent [-]

The concern is that attestation turns into another mechanism to monitor and control people; see how age verification turns into surveillance.