This is so true. Not making excuses for Microsoft, but OP’s at this very hard. What would have been a “reboot the server” at small scales becomes a concerted operational nightmare at larger scales.

Why so? I'd have thought that in situations like this, one would limit inbound connections, either to certain IPs, or through rate limiting controls, all of which should be controllable at the router / network infrastructure level.

Even more so when dealing with a potential incursion (like servers compromised with botnet software that goes back to work not long after a reboot and slams the server and/or network.)

It's very common, but typically teams test things by having a staging instance which also has staging versions of all dependencies.

They then deliberately loadtest it to failure and check it can be bought back under control.

See also the electrical grid.

I think this has to do more with their calculating SLAs for uptime than anything else. Once they have a root cause and start deploying a fix they can "stop the clock" on the outage, even if they don't get to your tenant right away.

Imagine the amount of money they're losing right now in credits that will have to be applied once this finally gets resolved.

> I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available.

If nothing else, increased change volume will mean the number of candidate diffs ballooned, possible to a level where non-AI review is impossible on incident-level timescales.

Coupled with each change also being AI-documented and the relevant teams having less hands-on understanding of their own changes, so going to the pushing team and asking "Tell me about _____" isn't an option.

AI is at fault for what seem like some of the most vaguely worded outage stuff I have seen.

The email outage gives me time to order popcorn off Amazon for the inevitable postmortem in a few days. I hope it's a good one!

You can be surprised because it (apparently) had nothing to do with AI or release pressure. It was a combination effect of two separate network changes.

> Nobody has ever fully understood how their systems work. Thinking largely about "systems as imagined" and "systems as found."

But with AI, now we can have systems that no one has understood ever.

thanks for this. Just read it for the first time. 1909 hits above its weight!