Because there's no indications that it is the case, at least for the time being.

No, my family is in Europe and Australia and also seeing it.

How would that help.

The issue now is fixing this so status of what happens is needed and not how it happened in the first place.

The cause will ans should only be published when the issue is fixed and reviewed.

It doesn't look like a cyberattack at all, talking about fixing health checks and the like looks much more like broken automation.

Australia affected as well.

Don't attribute to malice that which can be explained by stupidity.

Japan is also seeing the issue. It is probably a global issue.

Impacting us here in Australia, our company is having fun today.

Or a training run gone awry?

Nope. Europe (eastern) for sure included too.

[dead]

[dead]

[dead]

Considering the middle management culture at MSFT and other tech BigCo's it's probably better that this be a huge public outage where their crappy managers will actually face some heat for creating the kinds of processes that lead to these outcomes rather than can throw the team under the bus when inevitable human error creeps into their obviously un-followable at scale over time process.

I hope they all quit cause microsoft fuckin sucks.

This needs to be amplified. A lot of the updates carry all the hallmarks of leadership consistently denying the engineering teams the time to build out sufficient introspective and troubleshooting tooling, on the finance-supported rationale that it is not financially justifiable for such low probability events regardless of impact. And this can be as simple as allowing for enough shadow non-Production environments real-time twinning inputs at scale enough to reproduce thundering herd issues, for example.

Take 8:52 PM UTC, for instance. "While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly." So many questions flow from this.

They jumped from "affected servers have become unresponsive" straight to "restarting these servers". No granular troubleshooting of a random sample of the unresponsive cohort to identify the precise origin of the failure mode. Possibly their own organizational chaos prevents the team at the low granularity layer seeing the pattern of whole groups of servers, from effectively interacting with the right team to troubleshoot at the individual server, high granularity layer. I've seen that at many of my clients; sometimes the different teams at different granularity layers don't even have established engagement channels for when things blow up across those granularity layers. I could go on, but you get the drift.

If you obsess over internal tooling, you waste time yak shaving and ship nothing; if you only care about shipping features, technical debt accumulates until a major outage hits. Because cutting corners for speed is the path of least resistance, paying down that complexity later is much harder than maintaining discipline upfront.

Hopefulness continues, I guess?

From the 2:14:51 AM UTC update:

Current status: We're applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We're continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.

---

Hopefulness returns.

From the 1:50:27 AM UTC update:

Current status: We're seeing positive results with our efforts to restart affected infrastructure. We're actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.

While I think it's a bit premature to jump to "AI is at fault", and I will emphasize that I appreciate Microsoft's transparency in their status updates, these updates do have much more of a "we don't know what's going on" feel than any other major outage I can think of.

It's been hours, and the status updates still feel like there is an undertone of frantic panic (understood if you can't determine the root cause). When I think of past major outages, they may have been down for hours but usually folks knew what the root cause was relatively early, even if remediation took some time.

I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available.

"Everything went down, trying to bring back a subset of things gets killed by thundering herd" is very common.

Nobody has ever fully understood how their systems work. Thinking largely about "systems as imagined" and "systems as found."

Richard Cook wrote about this years ago, and it's always going to be relevant:

https://highscalability.com/designing-for-resiliency-will-be...

https://how.complexsystems.fail/

Wonder why.

E.M. Forster says hello.

https://americanliterature.com/author/em-forster/novella/the...

Reading the status updates feels a lot like watching some of the OpenAI models try to debug a thorny issue. Throwing half-baked solutions at the wall, then reacting in surprise when it makes things worse, getting increasingly detached from reality until someone clears the context and makes it start over.

[dead]

[flagged]

I suspect that they are not. Copilot is also down :O

My girlfriends work she is hearing this from office employees. Kind of scary how much the public gets wrong.

She uses outlook.

It's not a bad solution, it definitely should be tried. But I'm still rather shocked to see it in official notes, especially this "late" in the timeline.

In rail, we call it BOBO - battery off, battery on.

If they rebooted it, and it started working again, would you just walk away? Or would you try to figure out why the fuck it needed to be rebooted in the first place?

Yeah. It works, almost always: https://en.wikipedia.org/wiki/Indonesia_AirAsia_Flight_8501

Rebooting is also the only solution for some problems.

looks at the Boeing 787

What makes sense for something that is used for 20 hours at a time is different from what makes sense for a service that literally has "365" in its name.

(on the ground) is probably pretty key, though. Ha ha. I can imagine the conversation if this was in the air.

I suppose you might want to first rule out a break-in, and if there is one, recover evidence that won't survive a reboot.

Yeah, a place I worked at years ago had a cron job to restart some critical service at midnight because it leaked memory all day. That was there for years.

~25 years ago I was a firewall administrator, and the Cisco PIX we used was notorious for gradually running out of memory, effectively cutting off the entire company from the Internet when it went down (and it was a very big company).

We absolutely should have scheduled reboots during off hours. Lessons learned.

the layman understands both of those words and their meaning together.

we "bounce" the servers when on calls with customers.

I have lived through and debugged cascading failures and large scale distributed systems.

The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.

Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).

A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.

We learned the hard way that there's a very non-obvious inflection point in large distributed systems, where the your design needs to change from accommodating load resiliently via queueing, auto scaling, and such, to a posture of actively scaling back certain features and even shedding load when in a defensive situation.

It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.

The best defense against cascading failures is limiting the blast radius, which is something AWS seems to do well but Azure does not.

Except for the fact that it seems like most Microsoft outages are self owns. I'm not sure how many times we've had outages due to bad deployments that somehow made it to their global infrastructure and only got noticed when customers notified.

Plenty of internal accounts from previous employees highlight this.

so, Bedlam 4?

Many of us insist upon not building Rube Goldberg machines that cannot be reasoned about precisely for this reason.

Most of the time we get demonized by resume-driven developers job-hopping every few years (seems like the lack of deep expertise is working out). Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.

I'd start with "what has changed" but surely they've looked at that.

There was an article posted by a former MS / Azure person earlier this year that described the dumpster fire of node management and their cloud control plane

https://isolveproblems.substack.com/p/how-microsoft-vaporize...

HN discussion - https://news.ycombinator.com/item?id=47616242

To be fair: I want the people who can speak most competently to a problem to be working on it versus writing commentary. Those highly-technical people can shine in the postmortem report.

And that is a good thing. Well, maybe not good. But the last thing you need in a crisis is to pull people away from fixing the problem and make them type up status updates. Admittedly, having tech knowledge would help the writers - a technical incident management team would be better. But ain't nothing wrong with letting the PR folks write up words while the tech folk work on problem solving.

Actual tech write-ups can come later.

Professional hour would be invisible. Outages don't happen because they were anticipated and prevented.

You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.

I'm not thrilled with what they've done to Windows lately myself, but this is just rude.

You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.

Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.

Don’t worry, I’m sure it’ll get better after they layoff 60,000 people.

A few weeks ago I saw a clip of a lady walking us through a day at her Microsoft life.

Between breakfast, 1-hour lunch, 15-min ice cream break, 30-mins chill, 30-mins foosball games I think she worked 4 hours max.

At that point she caught some flak since it was timed with the layoffs. And I get that is not necessarily representative of the MS culture…

… but talk about timing and poor choices!