Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.

The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.

▲

PaulHoule 3 hours ago | parent [-]

What gets me is how much the people who advocate for the "open web" have completely incorporated the values of Facebook and Google.

Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.

Another web is possible but web populists don't really believe it.

Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.

CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.

▲

armchairhacker 10 minutes ago | parent | next [-]

As others point out, the anime girl must exist because otherwise the sites can’t afford to serve you, amidst a new pattern of DDoS that is attributed to AI but I see as just DDoS becoming more prevalent (remember how sites used to occasionally be taken down by DDoS? Before Cloudflare (and Anubis) they’d just have to stay offline until the DDoSers got bored of wasting money for no good reason. Today’s “scrapers” apparently have new reasons that make them never bored of wasting money, but cannot be good enough to justify the aggressiveness that makes them DDoSers).

Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.

▲

dspillett 2 hours ago | parent | prev | next [-]

You seem quite incensed by the anime girl of Anubis. Unfortunately for you it is here to stay for a while, along with cloudflair's implementation and others, as they do reportedly make a difference to the sites using them as shields. Until something better is found, the other option is that the resources otherwise serving more bot traffic than they can afford simply go dark which would be worse.

Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.

> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit

They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.

> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits

I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.

▲

kg 2 hours ago | parent | prev | next [-]

What's the alternative? Invisible algorithmic fuzzy recognition on the server side that just randomly bans you if you look like a bad bot, with no recourse? At least if you solve a captcha and dismiss the cookie banner you can access the website.

▲

creata 2 hours ago | parent | prev [-]

> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.

Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.

> Another web is possible but web populists don't really believe it.

How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?

▲

dspillett 2 hours ago | parent | next [-]

> Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.

The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.

Memory hard challenges will reduce the effectiveness of this, but not remove it completely.

Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.

▲

kotaKat 2 hours ago | parent | prev | next [-]

Or they could just be more 'powerful' compute that can clear the challenge faster.

I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.

And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.

	▲	jon128 an hour ago \| parent [-]
		The "you need to enable cookies to view this site", cookies that will be limited to a browser-container and will be thrown away when the browser is closed anyway, is a sign that this modern nu web is broken in a fundamental way.

▲

jon128 an hour ago | parent | prev [-]

> They can't "async" their way out of that work.

Sharing cookies along a strata of inter-team bots is one trick.

Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".

> What alternative web could change this state of things?

A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.

IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.