I feel bad for the people working on this. It sounds like desperation is increasing.

From the 10:11:39 PM UTC update:

Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

From the 9:41:46 PM UTC update:

Current status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.

From the 9:01:14 PM UTC update:

Current status: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.

From the 8:44 PM UTC update:

Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.

Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.

▲

NolF a day ago | parent | next [-]

This seems like the same terse language/feedback OpenAI models give me and I leave more clueless than I arrived.

▲

baconbrand a day ago | parent | prev | next [-]

I hope everyone involved in working on this is staying grounded and relaxed and not suffering a ton of physical stress over what is ultimately the result of greedy leadership decisions :(

▲

cucumber3732842 14 hours ago | parent | next [-]

Considering the middle management culture at MSFT and other tech BigCo's it's probably better that this be a huge public outage where their crappy managers will actually face some heat for creating the kinds of processes that lead to these outcomes rather than can throw the team under the bus when inevitable human error creeps into their obviously un-followable at scale over time process.

▲

franktankbank 17 hours ago | parent | prev | next [-]

I hope they all quit cause microsoft fuckin sucks.

▲

NetMageSCW 12 hours ago | parent [-]

Better than most alternatives.

▲

Ygg2 11 hours ago | parent [-]

Say what you want about OpenOffice it doesn't experience 24h degradation because some idiot pushed vibe slop to production.

▲

EvanAnderson 9 hours ago | parent [-]

Not to defend Microsoft, but the thick-client Office applications (Word, Excel, PowerPoint, etc) worked fine during the outage. Nobody couldn't use Excel because of this outage. Outlook worked fine too if you were connecting it to an on-premises Exchange instance.

This was an outage of a server-hosted database for email, not an outage of desktop application software.

	▲	Ygg2 8 hours ago \| parent [-]
		Sure. Just wait till it gets fully entshittified. With thin clients you can enjoy any MSFT fuckup as free time off!

▲

yourapostasy 17 hours ago | parent | prev [-]

This needs to be amplified. A lot of the updates carry all the hallmarks of leadership consistently denying the engineering teams the time to build out sufficient introspective and troubleshooting tooling, on the finance-supported rationale that it is not financially justifiable for such low probability events regardless of impact. And this can be as simple as allowing for enough shadow non-Production environments real-time twinning inputs at scale enough to reproduce thundering herd issues, for example.

Take 8:52 PM UTC, for instance. "While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly." So many questions flow from this.

They jumped from "affected servers have become unresponsive" straight to "restarting these servers". No granular troubleshooting of a random sample of the unresponsive cohort to identify the precise origin of the failure mode. Possibly their own organizational chaos prevents the team at the low granularity layer seeing the pattern of whole groups of servers, from effectively interacting with the right team to troubleshoot at the individual server, high granularity layer. I've seen that at many of my clients; sometimes the different teams at different granularity layers don't even have established engagement channels for when things blow up across those granularity layers. I could go on, but you get the drift.

If you obsess over internal tooling, you waste time yak shaving and ship nothing; if you only care about shipping features, technical debt accumulates until a major outage hits. Because cutting corners for speed is the path of least resistance, paying down that complexity later is much harder than maintaining discipline upfront.

▲

NetMageSCW 8 hours ago | parent | next [-]

The fact that they don’t report certain steps in their public communications during an outage does not mean those steps weren’t done.

	▲	yourapostasy 2 hours ago \| parent [-]
		Fair point. Maybe at their scale they’ve learned something different about crisis management than I’ve experienced on a smaller scale. Would love to hear from those who have been in the trenches at a similar scale as these past couple of days. In a 1-2 hour unplanned outage of typically way smaller systems at some of my clients, I’m generally able to leave my status reports at this level we see here. But as the time and blast radius expands, I find it builds more trust that we’re doing everything within our power and builds more reassurance by sharing progressively more detail. The increasing detail also unlocks massive offers of assistance from many areas of the organization in many cases. Telling stakeholders what my team and I have tried and ruled out often helps them explain to their stakeholders just as much as telling them what has worked.

▲

cucumber3732842 14 hours ago | parent | prev [-]

Everyone wants to blame the bankers and the bean counters. As I get older I realize that businesses will happily forgo insane sums in the name of predictability and continuity and that it's the team leads and the middle managers who don't want the troubleshooting and the control panels because those would provide an quasi-audit log that would more readily quantify bad management. They see these systems as a threat, a way for information of what's really happening to leak around their control.

Of course not all team leads and middle managers are like this but too many of them are and the bad ones are a great fit for exploiting BigCo environments so BigCos act like a catch basin for them.

▲

EvanAnderson a day ago | parent | prev | next [-]

Working in reverse-chronological order here:

From the 1:06:17 AM UTC update:

Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery.

The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.

---

The list of impacted services just grew tremendously.

From the 12:56:09 AM UTC update:

Features of the Microsoft 365 services below that interact with Exchange Online and email services are impacted. Elements of those services that do not depend on interaction with Exchange Online remain functional.

- Exchange Online: Users may be unable to access their mailbox using Outlook on the web, Outlook desktop (MAPI), Exchange ActiveSync (EAS), or REST-based connections, and email delivery may be delayed.

- Microsoft 365 admin center: Admins may be unable to load some admin center pages.

- Microsoft 365 apps: Users may be unable to view their recently used files.

- Microsoft 365 Apps Admin Center: Multiple services including Cloud Update, Cloud Policy, and Inventory are affected.

- Microsoft 365 Education: Features including Assignments, Reading Progress, and Speaker Progress may be impacted.

- Microsoft Copilot 365: Users may be unable to load Copilot Chat or receive responses in the web and desktop apps, and Copilot experiences may be unable to access work data such as email, calendar, and files.

- Microsoft Defender for Office 365: Safe Links may be unable to verify URLs, which may impact users clicking links in Outlook and Teams.

- Microsoft Defender XDR: Users may be unable to load Microsoft Defender portal pages or solution tabs.

- Microsoft Fabric: Users may encounter failures when exporting reports with sensitivity labels to PDF or PowerPoint, or when updating artifacts with sensitivity labels.

- Microsoft Graph: API requests that access Exchange Online data, such as mail, calendar, and contacts, may time out.

- Microsoft Intune

- Microsoft Planner: Users are experiencing Plan and Task operation reliability issues.

- Microsoft Purview: Multiple Purview services including Authorization, auto-labeling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted, and users may be unable to load Microsoft Purview portal pages.

- Microsoft Teams: Scenarios that depend on Exchange Online, including calendar, meeting scheduling, and creating chats for organizations using Information Barriers, may fail. Core messaging and online meetings remain largely functional.

- Microsoft To Do: The app may not load, and users may see operation failures.

- Power BI: All users may intermittently be unable to export reports with MIP sensitivity labels to PDF in Power BI.

- SharePoint Online: Features that depend on Exchange Online may be impacted. Accessing files stored directly in SharePoint Online remains functional.

- Universal Print: Users may encounter errors submitting print jobs or using admin features.

- Windows 365: Customers who have configured their tenant and Cloud PCs to use Microsoft Purview Customer Key for Windows 365 Cloud PCs may have users who are unable to connect to their Cloud PCs.

Current status: We've reviewed known impact scenarios and have revised the list of impact services and features, which can be viewed in the "more info" section of this communication.

We're continuing our work to isolate the issue affecting server availability, as well as other workstreams.

---

So much for seeming hopeful.

From the 12:08:53 AM UTC update:

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.

---

The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.

From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

▲

EvanAnderson a day ago | parent [-]

Hopefulness continues, I guess?

From the 2:14:51 AM UTC update:

Current status: We're applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We're continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.

---

Hopefulness returns.

From the 1:50:27 AM UTC update:

Current status: We're seeing positive results with our efforts to restart affected infrastructure. We're actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.

▲

nwallin a day ago | parent [-]

They've tried turning it off and back on again, now they're moving on to identifying strategic actions that can help improve service health.

This doesn't sound like hopefulness, it sounds like somebody's huffing hopium. It sounds like somebody's who's all out of ideas, but there's no direction to go but forward.

	▲	a day ago \| parent [-]
		[deleted]

▲

matt3210 a day ago | parent | prev | next [-]

Sounds like nobody knows how their own systems work anymore.

▲

adamsb6 a day ago | parent | next [-]

"Everything went down, trying to bring back a subset of things gets killed by thundering herd" is very common.

▲

Arbortheus 19 hours ago | parent | next [-]

This is so true. Not making excuses for Microsoft, but OP’s at this very hard. What would have been a “reboot the server” at small scales becomes a concerted operational nightmare at larger scales.

▲

gordonfish 12 hours ago | parent | prev | next [-]

Why so? I'd have thought that in situations like this, one would limit inbound connections, either to certain IPs, or through rate limiting controls, all of which should be controllable at the router / network infrastructure level.

Even more so when dealing with a potential incursion (like servers compromised with botnet software that goes back to work not long after a reboot and slams the server and/or network.)

	▲	adamsb6 10 hours ago \| parent [-]
		Some form of demand control is generally how you work around these problems, but maybe those parts are falling over too. For really big services it's also hard to exercise these processes on anything approaching actual prod scale. Ideally you can chop the architecture into pieces where you can simulate region failure and ensure everything works as planned, and fix whatever you find that breaks with every simulation. But maybe some bits of the architecture aren't amenable to being regionalized.

▲

londons_explore 15 hours ago | parent | prev | next [-]

It's very common, but typically teams test things by having a staging instance which also has staging versions of all dependencies.

They then deliberately loadtest it to failure and check it can be bought back under control.

▲

voidfunc 14 hours ago | parent [-]

The staging pattern doesn't scale for systems this large and complicated. Everything is constantly in flux and changing. Ive worked on systems this large and we had something we called staging but it was a joke.

Gradual rollout, pilot regions and monitoring are usually how you avoid broken stuff becoming global incidents. Will be interesting to read what process failed here next week.

	▲	londons_explore 10 hours ago \| parent [-]
		This type of loadtest I normally only perform ~annually in the companies I have done it. It's normally a ~2 week-long project for an intern to identify failure modes of any client side apps (ie. Retry loops etc), capture a sample of production API requests, create some fake data store either on staging or an instance setup specially for the purpose, adjust the request sample to hit that fake data (reads, writes etc), then replay the sample faster and faster till staging fails due to overload, and practice a recovery procedure with the load still applied. If it goes well, redo it with different types of overload - especially large percentages of failure responses from various dependencies, or 1000ms delays added to dependency responses which tends to be one of the trickier things to recover from. Typical outcomes from the work are patches to make the service more robust, fallbacks to not use dependencies or disable features when overload happens, on-call procedures for what to do or not do in certain common cases etc.

▲

dboreham a day ago | parent | prev [-]

See also the electrical grid.

▲

quaffcode 16 hours ago | parent [-]

Betting the electric grid has people around who at least understand how their substation works.

▲

venzaspa 16 hours ago | parent [-]

It doesn't matter if large portions of the grid go down at once. See the recent Spanish outage - the coordination of bringing it back on line is an art of its own.

	▲	01HNNWZ0MV43FF 13 hours ago \| parent [-]
		For the curious: https://en.wikipedia.org/wiki/Black_start

▲

hn_throwaway_99 a day ago | parent | prev | next [-]

While I think it's a bit premature to jump to "AI is at fault", and I will emphasize that I appreciate Microsoft's transparency in their status updates, these updates do have much more of a "we don't know what's going on" feel than any other major outage I can think of.

It's been hours, and the status updates still feel like there is an undertone of frantic panic (understood if you can't determine the root cause). When I think of past major outages, they may have been down for hours but usually folks knew what the root cause was relatively early, even if remediation took some time.

I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available.

	▲	Root_Denied 19 hours ago \| parent \| next [-]
		I think this has to do more with their calculating SLAs for uptime than anything else. Once they have a root cause and start deploying a fix they can "stop the clock" on the outage, even if they don't get to your tenant right away. Imagine the amount of money they're losing right now in credits that will have to be applied once this finally gets resolved.
	▲	ethbr1 14 hours ago \| parent \| prev \| next [-]
		> I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available. If nothing else, increased change volume will mean the number of candidate diffs ballooned, possible to a level where non-AI review is impossible on incident-level timescales. Coupled with each change also being AI-documented and the relevant teams having less hands-on understanding of their own changes, so going to the pushing team and asking "Tell me about _____" isn't an option.
	▲	Aloha a day ago \| parent \| prev \| next [-]
		AI is at fault for what seem like some of the most vaguely worded outage stuff I have seen.
	▲	a day ago \| parent \| prev \| next [-]
		[deleted]
	▲	gred a day ago \| parent \| prev \| next [-]
		The email outage gives me time to order popcorn off Amazon for the inevitable postmortem in a few days. I hope it's a good one!
	▲	NetMageSCW 8 hours ago \| parent \| prev \| next [-]
		You can be surprised because it (apparently) had nothing to do with AI or release pressure. It was a combination effect of two separate network changes.
	▲	15 hours ago \| parent \| prev [-]
		[deleted]

▲

jacquesm 7 hours ago | parent | prev | next [-]

Wonder why.

▲

riffic a day ago | parent | prev | next [-]

Nobody has ever fully understood how their systems work. Thinking largely about "systems as imagined" and "systems as found."

Richard Cook wrote about this years ago, and it's always going to be relevant:

https://highscalability.com/designing-for-resiliency-will-be...

https://how.complexsystems.fail/

	▲	10111001 a day ago \| parent [-]
		> Nobody has ever fully understood how their systems work. Thinking largely about "systems as imagined" and "systems as found." But with AI, now we can have systems that no one has understood ever.

▲

netsharc a day ago | parent | prev | next [-]

E.M. Forster says hello.

https://americanliterature.com/author/em-forster/novella/the...

	▲	yes_m_ai_n a day ago \| parent [-]
		thanks for this. Just read it for the first time. 1909 hits above its weight!

▲

solid_fuel a day ago | parent | prev | next [-]

Reading the status updates feels a lot like watching some of the OpenAI models try to debug a thorny issue. Throwing half-baked solutions at the wall, then reacting in surprise when it makes things worse, getting increasingly detached from reality until someone clears the context and makes it start over.

▲

imposter 21 hours ago | parent | prev | next [-]

[dead]

▲

ajdude a day ago | parent | prev [-]

[flagged]

▲

fock a day ago | parent | prev | next [-]

their employer brought me agent swarms supposed to take over my php-serving machines. whatever this is, I don't feel sorry for them.

▲

trebligdivad a day ago | parent | prev | next [-]

Nah, they're just debugging a hard problem while the world looks on! The 'normal' problems they have processes to work through, but hey emergent systems and all that - they have to expect the unexpected.

▲

eddythompson80 a day ago | parent | prev | next [-]

they are furiously typing at their agents hoping they can figure out what to do. Getting glimps of hope everytime claude says "I found it." only for their hopes to be crused at "Turns out ...". The clown show at Microsoft is surreal.

▲

Rantenki a day ago | parent | next [-]

I suspect that they are not. Copilot is also down :O

	▲	tharkun__ a day ago \| parent [-]
		That's why your parent said "their CLAUDE" ;)

▲

a day ago | parent | prev | next [-]

[deleted]

▲

a day ago | parent | prev [-]

[deleted]

▲

piker a day ago | parent | prev | next [-]

So the thundering herd wins again

▲

reaperducer a day ago | parent | prev [-]

Ugh. There's some yahoo from NewsNation on WGN Radio right now saying this is because of the Chinese, and obliquely linking it to Nvidia.

Reddit-level speculation.

▲

llama052 a day ago | parent [-]

My girlfriends work she is hearing this from office employees. Kind of scary how much the public gets wrong.

▲

b0rtb0rt a day ago | parent [-]

and what evidence do you have that they’re wrong?

▲

aquova a day ago | parent [-]

What evidence do you have that they're right?

	▲	b0rtb0rt 5 hours ago \| parent \| next [-]
		gp confidently claims that people are wrong but no one really knows for sure, i doubt they would publicly announce that it’s a cyberattack yet
	▲	ethbr1 14 hours ago \| parent \| prev [-]
		This is 2026, why are we asking for evidence?