Also, for anyone who only has an old global API key lying around instead of the more recent tokens, you can set:

  -H "X-Auth-Email: $EMAIL_ADDRESS" -H "X-Auth-Key: $API_KEY"

Edit: and in case you're like me and thought it would be clever to block all non-Cloudflare traffic hitting your origin... remember to disable that.

This is exactly what we've decided we should do next time. Unfortunately we didn't generate an API token so we are sitting twiddling our thumbs.

Edit: seems like we are back online!

Awesome! I did it via the Terraform provider, but for anyone else without access to the dashboard this is great. Thank you!

I would advise against this action. Just ride the crash.

Good advice!

And no need for -X GET to make a GET request with curl, it is the default HTTP method if you don’t send any content.

If you do send content with say -d curl will do a POST request, so no need for -X then either.

For PATCH though, it is the right curl option.

If anyone needs the internet to work again (or to get into your cf dashboard to generate API keys), if you have Cloudflare WARP installed, turning it on appears to fix otherwise broken sites. Maybe using 1.1.1.1 does too, but flipping the radio box was faster. Some parts of sites are still down, even after tunneling into to CF.

Because no one cares enough, including users.

Oddly this centralization allows a complete deferral of blame without you even doing anything: if you’re down, that’s bad. But if you’re down, Spotify is down, social media is down… then “the internet is broken” and you don’t look so bad.

It also reduces your incentive to change, if “the internet is down” people will put down their device and do something else. Even if your web site is up they’ll assume it isn’t.

I’m not saying this is a good thing but I’m simply being realistic about why we ended up where we are.

Many reasons but DDoS protection has massive network effects. The more customers you have (and therefore bandwidth provision) the easier it is to hold up against a DDoS, as DDoS are targeting just one (usually) customer.

So there are massive economies of scale. Small CDN with (say) 10,000 customers and 10mbit/sec per customer can handle 100gbit/s DDoS (way too simplistic, but hopefully you get the idea) - way too small.

If you have the same traffic provisioned on average per customer and have 1 million customers, you can handle a DDoS 100x the size.

Only way to compete with this is to massively overprovision bandwidth per customer (which is expensive, as those customers won't pay more just for you to have more redundancy because you are smaller).

In a way (like many things in infrastructure) CDNs are natural monopolies. The bigger you get -> the more bandwidth and PoP you can have -> more attractive to more customers (this repeats over and over).

It was probably very astute of Cloudflare to realise that offering such a generous free plan was a key step in this.

Yeah, I went to HN after the third web page didn't work. I am not just worried about the single point of failure, I am much more worried about this centralization eventually shaping the future standards of the web and making it de facto impossible to self-host anything.

Well that and the fact that when 99% goes through a central party, then that central party will be very interesting for authoritarian governments to apply sweeping censorship rules to.

Another one that worries me is Let's Encrypt.

It is not as bad as Cloudflare or AWS because certificates will not expire the instant there is an outage, but considers that:

- It serves about 2/3 of all websites

- TLS is becoming more and more critical over time. If certificates fail, the web may as well be down

- Certificate lifetimes are becoming shorter and shorter, now 90 days, but Let's Encrypt is now considering 6 days, with 47 days being planned as a minimum

- An outage is one thing, but should a compromise happen, that would be even more catastrophic

Let's Encrypt is a good guy now, but remember that Google used to be a good guy in the 2000s too!

This might sound crazy as a software engineer, but I actually like the occasional "snow day" where everything goes down. It's healthy for us to all disconnect from the internet for a bit. The centralization unintentionally helps facilitate that. At least, that's my glass half full perspective.

Mostly since the AWS craze started a decade ago, developers have gone away from Dedicated servers (which are actually cheaper, go figure), which is causing all this mess.

It's genuinely insane that many companies are designing a great amount of fallbacks... on the software level but almost none is thought on the hardware/infrastructure level, common-sense dictate that you should never host everything on a single provider.

They're using cloudfare for multicloud, but still have cloudfare as a single point of failure. Should make a cloudfare for cloudfare to solve this.

How did we get to a place where Cloudflare being down means we see an outage page, but on that page it tells us explicitly that the host we're trying to connect to is up, and it's just a Cloudflare problem.

If it can tell us that the host is up, surely it can just bypass itself to route traffic.

Now that network effects and data lock-in have taken root, downtime is not as big of a concern as it was in the 2000s

Totally cooked if you have Cloudflare fronting us-east-1, with no redundancies.

It's not only centralization in the sense your website will be down if they are down but it is also a centralized MITM proxy. If you transfer sensitive data like chats over cloudflare-"protected" endpoints, you also allow CF to transparently read and analyze it in plain-text. It must be very easy for state agencies to spy on the internet nowadays, they woukd just ask CF to redirect traffic to them.

I recommend this Ben Thompson piece on why resiliency has declined: https://stratechery.com/2025/resiliency-and-scale/

Because it's better to have a really convenient and cheap service that works 99% of the time, than a resilient that is more expensive or more cumbersome to use.

It's like github vs whatever else you can do with git that is truly decentralized. The centralization has such massive benefits that I'm very happy to pay the price of "when it's down I can't work".

Most developers don't care to know how the underlying infrastructure works (or why) and so they take whatever the public consensus is re: infra as a statement of fact (for the better part of the last 15 years or so that was "just use the cloud"). A shocking amount of technical decisions are socially, not technically enforced.

Slowly and with full conscience of where we were heading to.

It's weird to think about so bear with me. I don't mean this sardonically or misanthropically. But, it's "just the internet." It's just the internet. It dones't REALLY matter in a large enough macro view. It's JUST the internet.

Because DDoS is a fact of life (and even if you aren't targeted by DDoS, the bot traffic probing you to see if you can be made part of the botnet is enough to take down a cheap $5 VPS). So we have to ask - why? Personally, I don't accept the hand-wavy explanation that botnets are "just a bunch of hacked IoT devices". No, your smart lightbulb isn't taking down Reddit. I slightly believe the secondary explanation that it's a bunch of hacked home routers. We know that home routers are full of things like suspicious oopsie definitely-not-government backdoors.

For most services it's safer to host from behind Cloudflare, and Cloudflare is considered more highly available than a single IaaS or PaaS, at least in my headcanon.

This topic is raised every time there is an outage with cloudflare and the truth of the matter is, they offer an incredible service, there is not a bit enough competition to deal with it. By definition their services are so good BECAUSE their adoption rate is so high.

It's very frustrating of course, and it's the nature of the beast.

Short-term economic forces, probably. Centralization is often cheaper in the near term. The cost of designing in single-point failure modes gets paid later.

Agreed. More worrying is that it appears standard practice or separation between domain and nameserver administration has been lost to one-stop-shop marketing.

I would be less worried if Cloudflare and AWS weren't involved in many more things than simply running DNS.

AWS - someone touches DynamoDB and it kills the DNS.

Cloudflare - someone touches functionality completely unrelated to DNS hosting and proxying and, naturally, it kills the DNS.

There is this critical infrastructure that just becomes one small part of a wider product offering, worked on by many hands, and this critical infrastructure gets taken down by what is essentially a side-effect.

It's a strong argument to move to providers that just do one thing and do it well.

IMO, centralization is inevitable because the fundamental forces drive things in that direction. Clouds are useful for a variety of reasons (technical, time to market, economic), so developers want to use them. But clouds are expensive to build and operate, so there are only a few organizations with the budget and competency to do it well. So, as the market matures you end up with 3 to 5 major cloud operators per region, with another handful of smaller specialists. And that’s just the way it works. Fighting against that is to completely swim upstream with every market force in opposition.

Because they are great services, are generally pretty easy to get started with, and usually work as expected, which has led to broad adoption.

Compliance. If you wanna sell your SAAS to big corpo, their compliance teams will feel you know what you're doing if they read AWS or Cloudflare on your architecture, even if you do not quite know what you're doing.

This was always the case. There was always a "us-east" in some capacity, under Equinix, etc. Except it used to be the only "zone," which is why the internet is still so brittle despite having multiple zones. People need to build out support for different zones. Old habits die hard, I guess.

All decentralized systems tend to centralization over time.

Well the centralisation without rapid recovery and practices that provide substantial resiliency… that would be worrying.

But I dare say the folks at these organisations take these matters incredibly seriously and the centralisation problem is largely one of risk efficiency.

I think there is no excuse, however, to not have multi region on state, and pilot light architectures just in case.

> How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down?

As always, in the name of "security". When are we going to learn that anything done, either by the government or by a corporation, in the name of security is always bad for the average person?

The same reason we have centralization across the economy. Economies of scale is how you make a big business succesful, and once you are on top its hard to dislodge you.

Except businesses love it.

A lot (and I mean a lot) of people in IT like centralization specifically because it’s hard to blame people for doing something that everyone else is doing.

Consider joining the Internet Society. An entire group of people who care!

A lot of products use AWS because "we could build redundancy and multi-region if we need it" and then never build it.

Currently at the public library and I can't use the customer inventory terminals to search for books. They're just a web browser interface to the public facing website, and it's hosted behind CF. Bananas.

What is worrying is that distributed systems don’t seem to be that distributed in practice.

It's because single points of traffic concentration are the most surveillable architecture, so FVEY et al economically reward with one hand those companies who would build the architecture they want to surveil with the other hand.

Don't think there is anything wrong with a centralised service being down, you just make a conscious decision if you want that and can afford that?

People not being ready for cloudflare/[insert hyperscaler] to be possibly down is the only fault.

Don't forget the CloudStrike outage: One company had a bug that brought down almost everything. Who would have thought there are so many single points of failure across the entire Internet.

We take the idea of the internet always being on for granted. Most people don’t understand the stack and assume that when sites go down it’s isolated, and although I agree with you, it’s just as much complacency and lack of oversight and enforcement delays in bureaucracy as it is centralization. But I guess that’s kind of the umbrella to those things… lol

The technical term for it is a man in the middle. It’s better to call it what it is that way you aren’t fooled into thinking it’s not, because it is.

How many more of these until governments step in and take over "critical infrastructure"?

And all of these outages happening not long after most of them dismissed a large amount of experienced staff while moving jobs offshore to save in labor costs.

When there is an accident on the interstate we should blame the centralization of traffic and advocate for no more highways.

Very worrying indeed.

Two ways. Gradually, then suddenly.

People use CloudFlare because it's a "free" way for most sites to not get exploited (WAF) or DDoSed (CDN/proxy) regularly. A DDoS can cost quite a bit more than a day of downtime, even just a thundering herd of legitimate users can explode an egress bill.

It sucks there's not more competition in this space but CloudFlare isn't widely used for no reason.

AWS also solves real problems people have. Maintaining infrastructure is expensive as is hardware service and maintenance. Redundancy is even harder and more expensive. You can run a fairly inexpensive and performant system on AWS for years for the cost of a single co-located server.

There is this tendency to phrase questions (or statements) as "when did 'we' ".

These decision are made individually not centrally. There is no process in place (and most likely there will never be) that will be able to control and dictate if people decide one way of doing things is the best way to do it. Even assuming they understand everything or know of the pitfalls.

Even if you can control individually what you do for the site you operate (or are involved in) you won't have any control on parts of your site (or business) that you rely on where others use AWS or Cloudflare.

because efficiency trumps redundancy in the short term, which is all that matters in a super competitive environment.

…sneaks in Azure

Is avoiding single point of failure in anyone’s playbook? ¯\_(ツ)_/¯

Because bots are a real thing.

And it’s hard to protect against DDoS without something like Cloudflare.

Look at the posts here.

Even the meager HN “hug of death” will take things down

Re: Cloudflare it is because developers actively pushed "just use Cloudflare" again and again and again.

It has been dead to me since the SSL cache vulnerability thing and the arrogance with which senior people expected others to solve their problems.

But consider how many people still do stupid things like use the default CDN offered by some third party library, or use google fonts directly; people are lazy and don't care.

A key risk of monopolies is that they lead to monoculture SPoFs.

because cloudfare protection blah blah, until cloudfare is down itself and then you are back to "who watches the watchmen"

Hacking software or hardware is so old school.

The target these days is the user.

The make-believe worm.

It's not really. People are just very bad at putting the things around them into perspective.

Your power is provided by a power utility company. They usually serve an entire state, if not more than one (there are smaller ones too). That's "centralization" in that it's one company, and if they "go down", so do a lot of businesses. But actually it's not "centralized", in that 1) there are actually many different companies across the country/world, and 2) each company "decentralizes" most of its infrastructure to prevent massive outages.

And yes, power utilities have outages. But usually they are limited in scope and short-lived. They're so limited that most people don't notice when they happen, unless it's a giant weather system. Then if it's a (rare) large enough impact, people will say "we need to reform the power grid!". But later when they've calmed down, they realize that would be difficult to do without making things worse, and this event isn't common.

Large internet service providers like AWS, Cloudflare, etc, are basically internet utilities. Yes they are large, like power utilities. Yes they have outages, like power utilities. But the fact that a lot of the country uses them, isn't any worse than a lot of the country using a particular power company. And unlike the power companies, we're not really that dependent on internet service providers. You can't really change your power company; you can change an internet service provider.

Power didn't used to be as reliable as it is. Everything we have is incredibly new and modern. And as time has passed, we have learned how to deal with failures. Safety and reliability has increased throughout critical industries as we have learned to adapt to failures. But that doesn't mean there won't be failures, or that we can avoid them all.

We also have the freedom to architect our technology to work around outages. All the outages you have heard about recently could be worked around, if the people who built on them had tried:

- CDN goes down? Most people don't absolutely need a CDN. Point your DNS at your origins until the CDN comes back. (And obviously, your DNS provider shouldn't be the same as your CDN...)

- The control plane goes down on dynamic cloud APIs? Enable a "limp mode" that persists existing infrastructure to serve your core needs. You should be able to service most (if not all) of your business needs without constantly calling a control plane.

- An AZ or region goes down? Use your disaster recovery plan: deploy infrastructure-as-code into another region or AZ. Destroy it when the az/region comes back.

...and all of that just to avoid a few hours of downtime per year? It's likely cheaper to just take the downtime. But that doesn't stop people from piling on when things go wrong, questioning whether the existence of a utility is a good idea.

5 mins. of thought to figure out why these services exist?

Dialogue about mitigations/solutions? Alternative services? High availability strategies?

Nah! It's free to complain.

Me personally, I'd say those companies do a phenomenal job by being a de facto backbone of the modern web. Also Cloudflare, in particular, gives me a lot of things for free.

CAPITALISM

Are people really this confused?

It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom.

You NEED to phase config rollouts like you phase code rollouts.

Wish this could rocket to the top of the comment thread, digging through hundreds of comments speculating about a cyberattack to find this felt silly

Configuration changes are dangerous for CF it seems, and knocked down $NET almost 4% today. I wonder what the industry wide impact is for each of these outages?

Tell him it's worse than he thinks. He obviously brought the entire Cloudflare system down.

Well, you can never be sure that he didn't:

https://www.fastly.com/blog/summary-of-june-8-outage

Is there a word for that feeling of relief when someone else fucked up after initially thinking it was you?

The company where this colleague works? Cloudflare.

I woke up getting bombarded by multiple clients messages of sites not working, I shitted my pants because I've changed the config just yesterday. When I saw the status message "cloudflare down" I was so relieved.

Good that he worked it out so quick. I recently spent a day debugging email problems on Railway PaaS, because they silently closed an SMTP port without telling anyone.

Do you guys work at Cloudflare? Do you mind reverting that change just in case?

Plot twist: They work at Cloudflare

Did your colleague perhaps change the Cloudflare config again right now? Seems to be down again.

How do we know your colleagues changes didn't take down Cloudflare though?

You should tell him his config change took down half the internet.

Even pornhub is down becuase it uses clouflare.

You missed a great opportunity to dead-pan him with something like "No, Bob, not just our site, you brought down the entire Internet, look at this post!"