Microsoft will still charge you money for Exchange SE, and occasionally release security and other bug fixes, but “supported” is an extremely generous description of the state of the product.

Exchange should be well protected behind a firewall or 10 after all these years.

What this leaves is a comparison of whether on-premise exchange servers have been compromised or down compared to the cloud.

I would hope that enough hardening has happened over the last 20 or 30 years.

Yep, same here from Germany.

From the Health Status page:

"Microsoft Teams: Most of the core functionality including joining calls, meetings, group chats, and channel messages are not impacted. Scenarios that depend on Exchange Online, including Calendar related failures with scheduling, invites, viewing, and meeting edits. Creation of Live or Virtual events fail. Failures with Copilot responses for outlook, calendar, or M365 files, read history, or adding to new chats. Navigating to App chats, adding new contacts or syncing contacts in settings fail. Standard channels are working fine but users may experience the inability to create or delete private or shared channels."

I suspect that the search is broken because it relies on the exchange bits to find users.

Teams relies on Exchange API to autocomplete your search for colleagues, if you know someone's full email you should be able to start it that way.

> What is driving me nuts is that we still heavily rely on email.

That itself wouldn't be a problem - email is very decentralised.

The problem is that we thought that putting like 2 companies in charge of most email communication is a good idea.

Meanwhile we were very happy that we mainly rely on email. We were still able to send out urgent messages (like "sorry if we don't reply, Microsoft is down, you can reach us by [phone] or [backup email]") via SMTP and if we really needed to receive some email, we'd be able to set up a secondary MX record to a holding server in about an hour.

If we were mainly using Teams or whatever other "better than email" service and that had an outage, we'd actually be in the dark!

As the resident complainer about Microsoft lock-in, I was honestly quite surprised by how well we could work around this. Don't get me wrong, if it was Entra that fell over, we'd be thoroughly fucked, but Exchange wasn't all that bad.

What’s worse is how many things force an email second factor now. No choice to make a TOTP or to use a passkey (which, with biometrics securing it, is two friggin factors at once, and should be all we need for anything that isn’t nuclear secrets!!!! Sorry this just kills me with every SaaS product I have to “two-factor” to daily).

Your email is down? Cool now you can’t log into anything.

Email has been quite robust over the past~ 30 years. Exchange is just email by Microsoft so of course it'll be more brittle.

It’s interesting how relative this is, reading by the comments and stories around them. A previous coworker started computing in the punch card era and would tell stories of dropping your stack of cards and having to reorder them, using the printouts to debug, and so on. And helping develop what’s now known as fork() for processes (although it predated fork and he had his own name for it), and helping develop a commercial RDBMS before they existed all in a non tech industry career.

So when I read about Exchange Server to me it still feels quite modern.

Don't frame it as a negative. This is what it feels like to know things that other people don't know.

To test our code changes, we would check out a VM that was running Microsoft Windows Server edition which was a beefy VM that was running Exchange. It was a carbon copy of the server running in production.

The Exchange code base frankly was a behemoth. No one had a full understanding of how it worked E2E. I am just sharing how I understood it. I was only a lowly L63 btw.

I doubt it has something to do with being old, running an email server of any kind, let alone knowing how email works, let alone using email, have been all various abstractions to help make users into consumers who pay more than people who know how to build and run this infrastructure.

There's Skype for Business SE too. They've recently brought back the group chat feature.

This is common knowledge. Try googling EdgeTransport.exe and you should see the outdated documentation for Exchange Server on-Prem.

It’s a distributed system with thousands of servers with automatic update distribution and config change distribution. Bad changes can propagate across the entire system though Microsoft status updates say they didn’t see any change that seemed relevant. It could also be a bug across all the uniform code that somehow triggered. For example Cisco BE servers once had a firmware bug that after 32767 hours the fans would all run at 100% speed.

Microsoft product reliability has been a running joke since the company started launching products. Why would anything change just because they pivoted to managed services instead of shipping install media?

I don't know what OP means. The deployment pipeline takes weeks or the release process?

The latter is probably the case for most enterprises.

Getting releases deployed into the testing environments, much less production, requires coordination, testing, reports and sign-offs.

Of course this can normally be expedited or bypassed for incident management and critical fixes.

All I see today is middle managers using an e-mail outage as an excuse for not getting things done yesterday.

Pick. Up. The. God. Damned. Phone. And do your job.

It's not about outlook the mailing app. They called their SaaS e-mail Outlook, you can use it without using Windows, in the same way as Gmail. Pretty popular in the corporate world

A large majority of companies I've worked for over the last 30+ years used Outlook and Exchange Server, or now M$365, for email, even in those cases when my actual work was on Linux. The exception were those that used Gmail...

Many, many big companies do, and some universities also moved to MS365 when Google downgraded their free campus agreement, because they were already paying for MS365 for Office and that came with cloud storage and outlook.

> I must live in a bubble... the last time I heard of anyone using Outlook was when Scott McNealy was calling it LookOut! in the 90s. [...] In a ~35 year tech career I have never been exposed to, nor used Windows nor Outlook and have never met anyone who does.

Yes, then you probably at least work in a bubble.

In my experience, it's more common than GSuite in most companies outside of startups and big tech and is ubiquitous at schools, among many, many other entities.

Most of the Fortune 500?

Sounds like you live in the Silicon Valley bubble. I don’t have a stat but I would wager that most businesses (especially larger ones) run on outlook/exchange.

You have probably mostly worked for tech companies where business employees were not the primary workforce. When you think about it most other business generally have everyone needing a word, excel or PowerPoint. It bundles cleanly with the whole package.

Windows is still by far the most popular OS for employees at big companies. Using Microsoft's SharePoint/Outlook/etc is also extremely common.

I'm assuming your "surprise" isn't genuine.

Every non tech company uses outlook, basically.

It's interesting what kind of bubbles people live in.

You are absolutely living in a bubble. The vast vast majority of US companies are using both Windows and Outlook.

I’m with you. I took a job in finance at one point, reporting to the CTO. On the first day they showed me my desk, a windows machine. I had not used it since high school, and not seriously then. It just never occurred to me that you would have engineers use windows (all the trading systems were Linux).

The CTO had to pull a bunch of strings to let me run Linux.

Why is there not a single mention that this could be a cyberattack? Is the issue only affecting the USA?

Cortana, play My Heart Will Go - oh.

This thread is now complete. We can all go home now.

This seems like the same terse language/feedback OpenAI models give me and I leave more clueless than I arrived.

I hope everyone involved in working on this is staying grounded and relaxed and not suffering a ton of physical stress over what is ultimately the result of greedy leadership decisions :(

Working in reverse-chronological order here:

From the 1:06:17 AM UTC update:

Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery.

The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.

---

The list of impacted services just grew tremendously.

From the 12:56:09 AM UTC update:

Features of the Microsoft 365 services below that interact with Exchange Online and email services are impacted. Elements of those services that do not depend on interaction with Exchange Online remain functional.

- Exchange Online: Users may be unable to access their mailbox using Outlook on the web, Outlook desktop (MAPI), Exchange ActiveSync (EAS), or REST-based connections, and email delivery may be delayed.

- Microsoft 365 admin center: Admins may be unable to load some admin center pages.

- Microsoft 365 apps: Users may be unable to view their recently used files.

- Microsoft 365 Apps Admin Center: Multiple services including Cloud Update, Cloud Policy, and Inventory are affected.

- Microsoft 365 Education: Features including Assignments, Reading Progress, and Speaker Progress may be impacted.

- Microsoft Copilot 365: Users may be unable to load Copilot Chat or receive responses in the web and desktop apps, and Copilot experiences may be unable to access work data such as email, calendar, and files.

- Microsoft Defender for Office 365: Safe Links may be unable to verify URLs, which may impact users clicking links in Outlook and Teams.

- Microsoft Defender XDR: Users may be unable to load Microsoft Defender portal pages or solution tabs.

- Microsoft Fabric: Users may encounter failures when exporting reports with sensitivity labels to PDF or PowerPoint, or when updating artifacts with sensitivity labels.

- Microsoft Graph: API requests that access Exchange Online data, such as mail, calendar, and contacts, may time out.

- Microsoft Intune

- Microsoft Planner: Users are experiencing Plan and Task operation reliability issues.

- Microsoft Purview: Multiple Purview services including Authorization, auto-labeling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted, and users may be unable to load Microsoft Purview portal pages.

- Microsoft Teams: Scenarios that depend on Exchange Online, including calendar, meeting scheduling, and creating chats for organizations using Information Barriers, may fail. Core messaging and online meetings remain largely functional.

- Microsoft To Do: The app may not load, and users may see operation failures.

- Power BI: All users may intermittently be unable to export reports with MIP sensitivity labels to PDF in Power BI.

- SharePoint Online: Features that depend on Exchange Online may be impacted. Accessing files stored directly in SharePoint Online remains functional.

- Universal Print: Users may encounter errors submitting print jobs or using admin features.

- Windows 365: Customers who have configured their tenant and Cloud PCs to use Microsoft Purview Customer Key for Windows 365 Cloud PCs may have users who are unable to connect to their Cloud PCs.

Current status: We've reviewed known impact scenarios and have revised the list of impact services and features, which can be viewed in the "more info" section of this communication.

We're continuing our work to isolate the issue affecting server availability, as well as other workstreams.

---

So much for seeming hopeful.

From the 12:08:53 AM UTC update:

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.

---

The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.

From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

Sounds like nobody knows how their own systems work anymore.

their employer brought me agent swarms supposed to take over my php-serving machines. whatever this is, I don't feel sorry for them.

Nah, they're just debugging a hard problem while the world looks on! The 'normal' problems they have processes to work through, but hey emergent systems and all that - they have to expect the unexpected.

they are furiously typing at their agents hoping they can figure out what to do. Getting glimps of hope everytime claude says "I found it." only for their hopes to be crused at "Turns out ...". The clown show at Microsoft is surreal.

So the thundering herd wins again

Ugh. There's some yahoo from NewsNation on WGN Radio right now saying this is because of the Chinese, and obliquely linking it to Nvidia.

Reddit-level speculation.

Ah no that was a mistake on my part! RSS Feed was in EST and I misremembered the difference

Why do you think this is a problem for Eliza?

Well done sir

We already call it Microsoft 356.

I worked on multi-million dollar airplanes. If a pilot called me with an issue and hadn't rebooted the plane (on the ground), I wouldn't even talk to them. Rebooting isn't always a bad solution to technology.

In maintenance, sometimes the dumb solution is the best. The servers that ran amazon.com had a memory leak and were subsequently restarted. This hack continues for years because it was the most cost-effective thing to do.

Put another way, if they didn’t turn it off and on again, I would think less of them because that’s the first thing you should always do unless you know better

Powercycling is a designed method to bailout Cisco routers. If you botch a configuration command and lose connection, you can have someone powercycle it to restore you last good configuration. Its not just a joke it really works more often than I'd like to admit.

Amateur hour. Everyone knows professionals say "power cycling" instead to make it sound complicated.

copilot, make this doom and gloom situation sound better. assume you're using the voice of a friendly but intensely focused sysadmin. use only neutral language.

Amateur hour? Not sure if you've lived through one of these cascading failures, but they are hard to design against, hard to load/chaos test for, and hard to recover from. The key is that this is a dynamic load, not a static load; and it's normally amplified by uncontrolled retry behavior by clients.

In retrospect, the cause and cure are obvious to everyone.

Newest update says that wasn't the issue after all. They say relatively small loads are actually causing servers to become unresponsive and have to be restarted.

They still don't know why this "core process" is maxing out the processor for no reason.

This sounds suspiciously what occurred when I was working at a large motor sports company. We had a major release of new year models coming out on the website. The deployment went super smooth and then in about 30 mins the site was crashing. Since we were a huge MS client, almost everything we ran was all MS top to bottom.

Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.

CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.

This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.

I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.

The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.

The people writing these updates are very obviously not technical and are least one step removed from the actual work being done to fix this.

I can't stand it when I see armchair analysts making claims like this.

I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour

What would it take for it to be professional hour?

At Microsoft every hour is amateur hour.

Copilot hour!

I read opinions like those but then I worked with Azure at previous jobs a lot and never had any really big issues. On the other side, I just tried using AWS for a small Saas of mine and after setting up SES, I got a shady email, from non-defult support amazon domain that I need to send them scans of my personal documents. Link in that email was blocked by uBlock Origin, domain was not the same as other emails, not a single notification in AWS Support Console anywhere so I ignored it as phishing. Got suspended two weeks later, not a single human in sight within 48 hours of me making a ticket, I just gave up and moved emails to Cloudflare. If I had actual real clients at that phase I'd be pretty pissed. I genuinely don't know how people blindly trust AWS with everything these days, it's 2026 and there's still not even a hard billing limit built into AWS.

There are huge swaths of azure engineering teams that deploy things entirely manually via azure portal. Insane.

What is MBO?

> We use Azure at work and it’s wild how much stuff is just manual labor behind the scenes on their platform.

AWS is no different. Everything is on fire all the time, but customers usually don't notice.

I essentially said this in a sibling post whose parent asked why it seems like nobody knows the codebase anymore except my post was immediately flagged.

I think this is the logical conclusion of using copilot to write your stuff for the last two years. Nobody knows what it does and more and more bugs are being solved by just pointing the AI back to it again. Combine with the fact that so many people have been laid off and replaced with copilot I'm not surprised that the few who remain aren't quite sure how to solve it

And yet, for some reason, we have to _prove_ this is the case, as though in the history of time, engineering something has ever proven ok as a careless endeavor.

The sales have fallen off compared to the Playstation.

A quick google search turned up these global sales numbers: PS3 87.40 million, PS4 117.20 million, PS5 96.44 million. Xbox 360 85.73 million, Xbox One 57.96 million, Xbox Series X/S 35.68 million.

> Hadn't heard about Xbox

Indeed you’re seeing the consequences of their past decade+ of management

Massive layoffs and studio closures and mergers like Obsidian merged with Bethesda.

New Xbox to cost $1200-$1800.

Gamepass limited to 15 hours a month for Ultimate.

It'll probably be an app instead of its own thing in the future. For Pluton-enabled systems only of course.

insufficiently supervised AI on top of poorly architected sales and marketing driven decisions looks like it's a good recipe for a nice warm garbage fire

> Luckily most European politicians are corrupt and can easily be bought to prevent them from ditching Azure and Windows.

They're not corrupt (in a lot of cases), mostly just ignorant. Like, they don't understand the tech, and microsoft is safe, they're already using it so handing over large swathes of their infra to them seems reasonable.

MS have wonderful sales people, and this is why they do well with governments and large enterprises.

There's also a co-ordination problem, in that all of your counterparties (like your citizens, for instance) are familiar with MS and therefore it costs a bunch locally to move, for little short-term reward.

This is actually a place where (good) regulation could derisk a lot of these changes, but you'd need a committed group of politicians who can convince the civil servants to keep at it. It'll happen, but slowly and in starts over the next 5-15 years.

I am aware that the authentication subsystems have major structural problems which make finding things like exfiltration bugs in azure surprisingly common. I suspect the current incident is related. I'm hoping it's also caused by too much insufficiently supervised AI coding on poor architectural foundations, and that being caused by that will be obvious and hopefully become public. But that's just speculation on my part.

After the last couple of updates on entra, where you keep getting the "loading dots" to just pull up is insane. I still use Azure or PowerShell to do the task by the time it can "load up".

My take is the reason companies use Azure is lack of imagination. Like Service Now. Just horrible software, and thems that make the decision to buy it aren't the one's who have to use it.

Microsoft has not layed off half of its workers: 2021-2026: +23% employees 2024-2026: -2% employees

What changed is the productivity per employee: 2021-2026: +60% 2024-2026: +38%

To be honest, I also assumed your take on this at first. But after a bit of research I found quite a different reality!

I have the pro Microsoft copilot subscription at work just so I can search Outlook and teams since the native search doesn't turn up things when I search exact strings that are in the messages or use filters for other things like sender or recipient. Crazy to have to use an llm just to find an email

This really is the honest truth, every interaction I have with the MS platform is somehow worse than an actual dumpster fire. AI is the only thing keeping the ship afloat. Then again maybe that's the point?

> Fabric

I see what you did there.

Fabric is a weird one, as it's sortof a GUI over spark, but also PowerBI and lots of weird services.

It feels like a "data plaform" for people who've never used one before and only have the vaguest idea of what it means.

That being said, Spark is pretty good and we're running a lot of workflows on it (and probably building more on top, I suspect that I'll be much angrier about it in approximately 6 months or so).

I'm sure they would love to try, but Copilot is also offline according to the status page.

You don't understand, the agents aren't fixing the outage because those human engineers didn't make the infrastructure AI first. The only possible follow-up from this is that the faulty infra needs to be completely thrown away in favor of the vibecoded replacement.

I don't understand how basically everything has languished under Nadella. Apple doesn't sell MacOS, so it should be less incentivized to keep improving it, but it is way better than Windows and the gap keeps widening. If I didn't use MacOS, I would just use Linux.

Windows does always cater to backwards companility and making IT departments happy, so that explains some of the lagging technology and systems. I can't believe NTFS is still the default file systems. That's generations behind Apple's APFS.

The stock has popped, and that's what Wall Street cares about, but I could see a world where this chronic underinvestment and mismanagement basically implodes the entire company in the coming years.

I made the switch to Linux Mint and it’s wonderful and almost triples my battery life. I’m also in the process of switching to a phone with graphene os which runs like butter and feels so much more secure, safe and private.

>>Btw does everyone see how insanely crappy Windows 11 is compared to MacOS

This is just an anecdote, but I have the exact opposite experience. I'm a C++ games developer working on Windows 11 every day and other than the micro frustrations with stupid UI, the system just works. I know I can rely on it day to day. But I was also given a MacBook last year for our iOS ports and I don't understand how anyone considers it a serious operating system for work - it's full of bugs to a point where I started using AI to work around some of them. I thought maybe MacOS 27 would fix some of them and sure it did - but then it introduced a whole bunch of new ones. As a developer it's probably the worst OS you could pick - if you hate windows for whatever reason go straight to Linux, at least you have actual control over your OS compared to MacOS where basic things like spotlight are broken and there's no way to fix them other than add to the thousands of complaints on apple forums.

Yup.

Nadella's AI fumble is a true generational fuck up. They had OpenAI, they had complete access to GPT models before any one, they even had the first coding assistant concept.

Then due to typical clueless management and lack of taste and complete buffoonery, they just fucked it all up. I'm not complaining because I think everyone is better off, but it's just amazing to have witnessed that even when they luck out on a bet, they manage to fuck it up. Copilot started as a coding assistant, and before it could breath they destroyed the brand by shoving the name into literally everything without thinking or taste. Ignored the coding part, because that's lame, and instead focused on half assed Windows or Office apps and integrations that no one will ever use. Once claude code dominated the scene, they realized the major fuckup and tried to catch up, but it's too little too late.

Again, not complaining really.