There isn't too much of a difference from their normal behavior.

No, since they're simply too many. For an e-commerce site I work for, we once had an issue where some bad-actor tried to crawl the site to set up scam shops. The list of IPs were way too broad, and the user-agents way too generic or random.

Its not one IP to block. Its thousands! And they're also scatter through different ip networks so no simple cidr block is possible. Oh, and just for the fun, when you block their datacenter ips they switch to hundreds of residential network ips.

Yes, they are really hard to block. In the end I switched to Cloudflare to just so they can handle this mess.

Wouldn't it be trivial to just to write a uwf to block the crawler ips?

Probably more effective would be to get the bots to exclude your IP/domain. I do this for SSH, leaving it open on my public SFTP servers on purpose. [1] If I can get 5 bot owners to exclude me that could be upwards of 250k+ nodes mostly mobile IP's that stop talking to me. Just create something that confuses and craps up the bots. With SSH bots this is trivial as most SSH bot libraries and code are unmaintained and poorly written to begin with. In my ssh example look for the VersionAddendum. Old versions of ssh, old ssh libraries and code that tries to implement ssh itself will choke on a long banner string. Not to be confused with the text banner file.

I'm sure the clever people here could make something similar for HTTPS and especially for GPT/LLM bots at the risk of being flagged "malicious".

[1] - https://mirror.newsdump.org/confuse-some-ssh-bots.html

About 90%+ of bots can not visit this URL, including real people that have disabled HTTP/2.0 in their browser.

Maybe :-)

But for a small operation, AKA just me, it's one more thing for me to get my head around and manage.

I don't run just one one website or one service.

It's 100s of sites across multiple platforms!

Not sure I could ever keep up playing AI Crawler and IP Whack-A-Mole!

Of course :-)

And my image CDN blocked ByteSpider for me.

For a while I also blocked the entirety of Singapore due to all the bots coming out of AWS over there!

But it's honestly something I just dont need to be thinking about for every single site I run across a multitude of platforms.

Having said that, I will now look at the options for the business critical services I operate for clients!

Bad actors now have access to tens of thousands of IPs and servers on the fly.

The cost of hardware and software resources these days is absolute peanuts compared to 10 years ago. Cloud services and APIs has made managing them also trivial as hell.

Cloudflare is simply a evolution in response to the other side also having evolved greatly, both legitimate and illegitimate users.

> I think partially is not having to worry about certs is a nice reason to hide behind the proxy.

Use Caddy. I never worry about certs.

Don't you need a cert anyway to secure the connection from Cloudflare to your server?

> Issues are stable at this time. The targeted customer has implemented CloudFlare, and we have taken steps to mitigate this event.

I'm still confused. Does this mean that HN switches CF on or off in response to recent volume of bot traffic?

because we're an open source project that accepts pull requests on github and we'd like our PR submitters to see why their PRs are failing tests

I always see such negative responses when HN brings up software bloat ("why is your static site measured in megabytes").

Now that we have an abundance of compute and most people run devices more powerful than the devices that put man on the moon, it's easier than ever to make app bloat, especially when using a framework like Electron or React Native.

People take it personally when you say they write poor quality software, but it's not a personal attack, it's an observation of modern software practices.

And I'm guilty of this, mainly because I work for companies that prioritize speed of development over quality of software, and I suspect most developers are in this trap.

Merely reducing external dependencies causes people to come out in rashes.

A large proportion of “developers” enjoy build vs buy arguments far too much.

“We” are the ones making the architecture and the technical specs of these services. Taking care for it to still work when your favourite FAANGMC is down seems like something we can help with.

Or _you_ aren't down, but a third-party you depend on is (auth0, payment gateway, what have you), and you invested a lot of time and effort into being reliable, but it was all for less than nothing, because your website loads but customers can't purchase, and they associate the problem with you, not with the AWS outage.

Right. Whereas if we get whacked with a random DDoS, that's my fault.

Unfortunately Anubis doesn't help where my pipe to the internet isn't fat enough to just eat up all the bandwidth that the attacker has available. Renting tens of terabits of capacity isn't cheap and DDoS attacks nowadays are in the scale of that. BunnyCDN's DDoS protection is unfortunately too basic to filter out anything that's ever so slightly more sophisticated. Cloudflare's flexibility in terms of custom rulesets and their global pre-trained rulesets (based on attacks they've seen in the past) is imo just unbeatable at this time.

bunny.net is not reachable for me too... really funny

https://imgur.com/a/8gh3hOb

Why do people on a technical website suggest this? It's literally the same snake oil as Cloudflare. Both have an endgame of total web DRM; they want to make sure users "aren't bots". Each time the DRM is cracked, they will increase its complexity of the "verifier". You will be running arbitrary code in your big 4 browser to ensure you're running a certified big 4 browser, with 10 trillion man hours of development, on an certified OS.

That was possible when a DDos was usually still an occasional attack by a bad actor.

Most time I get ddosed now it's either Facebook directly, Something something Azure or any random AI.

So accept that your customers won't be able to use your services whenever some russian teenager is bored? Yeah, good luck with justifying that choice.

99.999% availability is around 5 minutes or so of downtime per year.

> Providers need to be more aware of their global impact in outages

So you think the problem is they aren't "aware"?

> 30 minutes of downtime

> this is tenable as long as these services are reliable

do you hear yourself, this is supposed to be a distributed CDN. imagine if HTTP had 30 minutes of downtime a year.

and judging by the HN post age, we're now past minute 60 of this incident.

> especially for AWS

CF can be just as difficult if not more to migrate off of especially when using things like durable objects

And I got a 504 error (served by CloudFront) on that status page earlier. The error message suggested there may have been a great increase in traffic that caused it.