There is so much spam, abuse, and aggressive scraping that I've had to block:

  - AWS
  - Google Cloud
  - Azure
  - Alibaba Cloud
  - Oracle

I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.

Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.

It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.

* like leaked .env routes, brute forcing admin panels, and so on

Can you do something with fail2ban? I.e. after a small number of requests resulting in 401 or 404, or a series of requests that's too fast to be reasonably from a human, ban that address for 48 hours.

If a legitimate request later comes from that address, it will probably get through.

But not sure how well iptables handles checking every connection against a list of possibly hundreds of thousands of addresses.

Bots and spiders hang onto old URLs longer than I hold onto a grudge.

We had some SSR that followed up with JavaScript listeners to do interactive updates instead of loading a whole new page. But the team responsible didn’t add nofollow and some bots ignore it anyway. Some significant fraction of our cluster size was answering those requests. But one deployment someone broke the fuck out of those links and it showed up immediately in our 4xx error statistics and we rolled the change back within a week.

Three years later I was still seeing those busted URLs being submitted.

Then the fix for the nofollow got implemented and declared done but the flag never got made permanent. We were pretty peeved when someone learned that in some offhand comment in a conversation between teams.

We were already identifying “good” and “bad” bots at the border and telling them to fuck off, so all of this was coming from our supposed friends. And that’s when I realized that a company owning a search engine and a cloud service provider is a protection racket. Google cloud is selling you servers in order to be talked to by Google bots.

> I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.

You may as well silently drop the packets if they don't adapt.

> like leaked .env routes, brute forcing admin panels, and so on

That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.

Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.

Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.

> These clouds must get the abuse happening under their own roof under control.

Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?

Meta went absolutely haywire on one of my sites a couple months ago.

I’ve had to block similar on clients sites. Their search pages would start to get hammered with obvious agentic search spam. Searching for thing not even related to the site in all combinations of the queries. Some of the user queries contained potentially PII.

Is there a handy list of all of this that users contribute to?

This is a very smart thing to do! Thanks.

Why allow random requests coming from the clouds?

However, they could just use residential proxy botnets.

I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).

AWS is terrible. We had another tenant on AWS attacking us on AWS. AWS support took nearly a week of slowly bouncing around a case to do anything.

This is a very smart thing to do! Thanks.

Why allow random requests coming from the clouds?

However, they could just use residential proxy botnets.

Isn’t scanning for vulnerabilities illegal? Traditionally security researchers had to be very careful about such things