As someone who worked in infrastructure for a large global CDN, this sort of overload issues happen all the time when trying to bring infrastructure back online after an issue.

First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.

In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.

Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).

However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.

▲

Waterluvian a day ago | parent [-]

I know it’s not actually that easy but don’t they just have some gateway that’s rejecting 100% of requests and they very gently lower that intentional rejection percentage over time?

▲

cortesoft a day ago | parent | next [-]

In many cases, yet it is that simple.

However, in other cases it can be more complicated. For example, if the requesters are other internal services, they might not be behind a single gateway (although you should then have other controls you can use). Other times, it is not easy to block the RIGHT requests, because it is hard to distinguish which backend services will end up being hit by a request (a single request can trigger multiple calls to multiple services).

There is also the problem of catch up - if a system in a stable state is close to max capacity, it might never have the capacity to recover from a failure even if the entire system is restored. For example, imagine a log processing system that can just barely handle the normal peak load; if you have a failure where your producers locally store logs until they can send them upstream, you will have to clear that backlog, which requires additional capacity beyond a full steady state, which you might not have.

▲

macintux a day ago | parent [-]

And you never really get an opportunity to thoroughly test these failure & recovery modes until something catastrophic happens, at which point capturing detailed metrics and assessing the right sequence to perform the actions are both challenging at best.

	▲	cortesoft 11 hours ago \| parent [-]
		Yep. There is simply no load testing you can do when you reach certain levels of traffic. Nothing can generate the type of load you get when you have millions of end users making requests from thousands of different networks, on different equipment and network configurations.

▲

fhood a day ago | parent | prev [-]

It is exactly that easy as long as it isn't the gateway that is the problem.