Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.

▲

kfhfardin 8 hours ago | parent | next [-]

I think this is the point that has become a moral dilemma. Increasingly, in this agentic age tasks like research and web exploration that would be done by humans and even buy things from an website is now done through agents. While I sympathize with the main author about Ddos, but blocking every agent regardless of intent seems cruel. This would allow an MCP doing deep research(almost identical to human behavior) to move forward. But there should definitely be more work done system that focus on agent intent(i.e some agent trying to find the best price for vacation or scraper building alternatives to Travel Advisor).

▲

ryandrake 8 hours ago | parent | next [-]

Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent requests a 1KB file from their web site. They care that automated systems are sucking down TBs per day, all day, every day.

▲

coldbrewed 8 hours ago | parent | prev [-]

Making people actually read content doesn't strike me as cruel. Tedious, sure, but not cruel. What is cruel is seeing all of our collective individuality and artistic expression jammed into an LLM, sold back to us by the token, and forced into our lives by an economy increasingly skewed towards holders of capital.

	▲	kfhfardin 8 hours ago \| parent [-]
		Fair point. Again, I might be terribly and almost Icarus-like optimistic, but I still believe the push and pull between Big Tech and Indie AI builders would still lead to an equilibrium that makes the world a little more efficient and perhaps in time, more creative than AI slop. Let's hope Open Source AI opens instead of OpenAI so the internet is not feeding the capitalist machine of OpenAI and Anthropic.

▲

coldbrewed 8 hours ago | parent | prev | next [-]

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.

▲

kfhfardin 7 hours ago | parent | next [-]

Hmm, I am sure you know more than I do on the topic. At least to me, an amateur in the security space, would the action of the scraper not be closer to repetitive calls to tables instead of, i.e a more inquisitive agent doing research or booking who spends more time looking through, and surely that can be tracked? Again, on the moral side, I agree that if one comes with the other, it is quite dystopian. I myself am an ML Inference Engineer, so I guess interesting problems and interesting solutions always draw me in much like this.

	▲	coldbrewed 6 hours ago \| parent [-]
		Abusive crawlers use residential proxies to distribute crawl traffic between thousands of IPs, so that one agent that's making five requests blends in with the massive crawl. There is signal but there are ML inference engineers tuning their traffic to blend in, which requires that mitigations must get more sensitive. At the end of the day, the defensive side has an obligation to protect customers and well behaved agents are the first thing that gets swept up along with the bad actors.

▲

deckar01 7 hours ago | parent | prev [-]

It’s not that dystopian if you are given the choice to trade your attestation for more valuable service. We have been paying for the web by letting advertisers correlate our behaviors together for a couple decades now.

	▲	coldbrewed 6 hours ago \| parent [-]
		The concern is that attestation turns into another mechanism to monitor and control people; see how age verification turns into surveillance.

▲

tadfisher 8 hours ago | parent | prev | next [-]

Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.

▲

dclaw 8 hours ago | parent | prev | next [-]

It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.

▲

transcriptase 8 hours ago | parent | prev [-]

That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.