Im not so sure about the infinite resources part, more like a self inflicted austerity of resources - its a technology problem emergent from a organizational problem, emergent from a certain late stage mindset

Does having 400M users and 1B emails per day factor into your forgiveness?

I’m no fan Outlook and what they’ve done for the world of email, but if they’re Rolls Royce and this outage is an engine failure then that’s an expected level of service?

Or they’re Boeing and this a 787 Max. I’m on the fence.

Great insight. I would add that forcing function comes from the self, not from outside. The force is essentially the communication between the parts. When the communication breaks down, things fall apart, establishing a new kind of communication across the pieces to achieve equilibrium. The physical structures also have the same rule. They will crumble into a pile of lego bricks to achieve a new state of equilibrium and a simplified network of communication across the pile.

Today is sounding like one hell of an incentive.

tl;dr

https://en.wikipedia.org/wiki/XBill

Do you have official reference for this?

Thanks for the tip!

"Office 364" is apparently a trending term.

I couldn't specifically say since my admin.microsoft.com fails pukes on a wildcard certificate for azureedge.net.

Also, multiple bankers I'm speaking to about Anthropic's IPO can't send or receive work e-mails. So that's a thing.

I wonder whether this will help the already strengthening trend of some governments to unstick themselves from Microsoft (in particular their cloud).

I don't expect a fundamental shift but the boss having no access to his mails certainly seems like something I would bring up quite a bit.

Yeah I mean a lot of people are nonchalant about it here, but email for business is like telephones. Basic infrastructure that has an assumed near-zero downtime and expectations of fault tolerance.

It's such bedrock infrastructure that I've seen it used as fallback in the event of electronic trading infrastructure outage, brokers would accept specifically formatted emails as orders.

At this point it would probably be less impact to have phones go out globally because people would fallback to walking across the floor, using zoom/slack for simultaneous communication or email for more async.

Felt bad for the engineers that it had to happen on a Friday but for a multi day outage I guess it's a blessing in disguise.

Barring corona & year end holidays, this is probably the closest we can get to a global work holiday.

> ... but email is a backbone of the modern world

That's why most don't realize how big GMail is. Google has 1.8 billion GMail users, including a great many who are using their Google email professionally (no matter if it's a Google Workspace with their own domain or a redirection or just a plain GMail). Apple has 1 billion email users.

Microsoft 400 million and, granted, many businesses but it's not a fourth the number of email users GMail has.

And many, many, many people (real estate, delivery drivers, Uber and the likes) are using either Apple or GMail email from... Their phones.

It's a good thing it's only the "small" Microsoft email users base that is affected and not the 1.8 billion GMail users.

Who wants to bet that this is ultimately due to AI, either an action taken by AI or just general AI code rot coupled with developer knowledge decay?

Not so long ago, I was assigned to a project that had to be built on Azure due to client requirements. Because we were underhanded and it was an MVP, we'd elected to use PaaS as much as possible: Postgres Flexible Server, Static Web Apps, Easy Auth, etc. On paper, these all looked fine. They had lots of documentation, example projects, GitHub repos, blogs, roadmaps, etc. Then we started building and running into issues, and we realised a bunch of these offerings were basically unmaintained. There were features that had been stuck in "preview" status for literally years. The blogs and roadmaps were no longer updated, and issues were piling up in GitHub.

I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI.

Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences.

Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions.

WinUI 3.0, WinAppSDK, and related CsWinRT, C++/WinRT are in a ridiculous position six years after Project Reunion announcement, due to ever meagre resources.

XBox app keeps under delivering user experience, and bugs on patch Tuesdays are getting a joke by now.

All this in a four trillion valued company.

Mythical man month, maybe throwing everybody in the company at it isn't the right thing to do. But they must be in a panic, and their own communications internally may well be hampered, could be MS everything is out for MS as well as everybody else.

In our field, once you understand what the problem is, the problem is usually solved. Unless it's broken hardware, obviously

At least 1

The ones who actually knew how the system works were too expensive to retain, so now it's just some interns typing frantically into Claude Code.

> How many Microsoft employees are working on this?

all 3!

I wonder if it’s an AI death spiral: “the service is down. Diagnose the failure.” … Hello?

How may Microsoft employees are left anyway?

/s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait...

And which made a load of staff redundant in the last month.

This is HN. We should be asking why frontier models couldn't still dolve this! How many parallel agents does MS have working on this?

211 muted people on the Teams call, 2 guys trying to figure it out, an incident manager asking for an ETA and status update every so often.

I don't understand why they haven't asked copilot to fix it yet, make no mistakes.

the vibe coding group in charge of this product has decided to delete the production database and backups

Maybe Nadella is still trying to figure out which of the 8000+ sacked employees knew how the system works /s

When your most important contacts use outlook, it does not matter what you do. Sending and receiving email when almost nobody else is sending and receiving is only small help.

Some problems are not solved by individual choice. If 30-40% of your important contacts use service that is down, your ability to work goes down dramatically.

I have currently important 3 daily contacts down they use MS services extensively contacted and said they can't do much of anything because their work process and the process of their contacts is down.

I dunno...

Wouldn't you then need to keep up-to-date backups but ensure the backups still stay secure? Learn all about and generate SPF, DKIM, and DMARC records but still regularly get sent to the spam folder or IP blacklists if your recipient has a big-provider email? Deal with self-rolled 2FA or passkeys and secure recovery methods. If you're self-hosting at your house, you're at the whims of nature, infrastructure, and creatures not disturbing the machine. You are the one patching security holes, or if you run someone else's containerized or easy-setup mail server, making the decision between daily updates that'll catch supply chain attacks and potentially introduce instability or new security holes; or monthly/annual/never updates that virtually ensure you have a few CVEs active at all times; or the lead maintainer quits/dies and you watch it deteriorate while users groupthink how to migrate their mail to a different container.

Sorry for the pessimism. I can't suggest a better alternative. I have one account on Gmail (for impersonal communication like shopping) and another on Fastmail and occasionally reflect on how annoying it will be to lose access to either.

There probably are some very good reasons not to self host your email server but apparently my own, self hosted, email server's uptime over the last decade is now better than that of outlook.com :-D

Quick lets collect all options people have to self host! in addition to docker-mailserver im aware of

- https://github.com/stalwartlabs/stalwart

- https://github.com/mailcow/mailcow-dockerized

Who wants to bet that Microsoft is using slack to communicate now?

For the amount of uneeded emails about future emails about other emails, probably no real work was lost.

Same is Aus. Wife works for government, they got sent home early (typical gov that means 30 minutes before official clock off)

The biggest issue has been so many security and approval flows depend on email, so all processes have ground to a halt.

We're coming to the end of the Friday 13th business day in East Asia. Can confirm it caused chaos for the whole day.

Just the UK? Lol, you can add many, if not most government and private companies and educational institutions in the US and countless other countries to that list. The government and corporate world is grinding to a halt.

This observation applies to many other countries, too. Mmmm centralization is pretty fun!

Now's the time for those NHS fax machines to shine, I guess.

Citation needed.

Government I have no idea, but loads of businesses don't use Outlook - alternatives do exist. I've personally not used Outlook for well over a decade now.

Quick google AI mode check says:

- MS 365/Outlook: ~40%

- Google Workspace: ~25%

- Zoho: ~2%

And then a bunch of others - Rackspace, Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al

Outside of tech and disaster recovery, people _really_ don't understand that some redundancy--in this context--is a good thing.

The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.

Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.

I don't see a reasonable alternative here. Pay for another email/office suite provider, and duplicate all mails and files? Twice the cost and exposure surface just to cover 0.01% of downtime.

Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.

Email in a large part is just a tool for communication(and mostly low priority at that). Where I work, we have backups: Teams( same) provider, Cellphone ( different provider ) , Landline( different provider)

We had a situation recently where cellphones were down, we switched to landline.

You don't need a 1 to 1 mode backup for everything, just a failover plan.

Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.

The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.

Your other source quote is a direct copy of a tweet from 2021: https://x.com/rrbone_net/status/1420656486573805570

You don't think if it was a DDoS attack they would've identified that by now?

I am positive there are bare-metal, no shared tenancy, customers of MS already. Unclear if they are down too. While yes, these companies are massive, not being allowed to share infrastructure defeats the entire revenue-purpose of offering a cloud. The hardware in these servers is unbelievably expensive. You'd go back to everyone having an IT department and a few racks of servers for hardware configurations that meet their specific needs (which I'm totally okay with, and even rooting for!)

I'm an outsider who used to work at a different cloud provider, so take this with a grain of salt, but this doesn't smell like a limits and provisioning problem, per se. It's not necessarily like they just had 1 too many emails tip the scales for the entire world. Until today stuff was running okay. This was a monitoring and fail-over problem, at a comically tragic scale - a cascade failure. Architecturally they obviously did not design for a world where they could damage control failure at this scale. Failure should be expected and designed for at literally every layer. They cheaped-out on careful design in order to play catch-up building a public cloud offering and here we are.

They could (and should, and clearly don’t) architect the services like this internally. There will always be some single-points-of-failure, but minimizing those should be a top priority. Just from reading their status updates it’s clear most everything is sharing way too much in terms of infrastructure and configuration.

But even this requires someone high up in the organization to have the insight that reliability requires some level of “inefficiency” — ie retaining certain highly paid staff, duplication of effort, over provisioning of resources, etc.

The irony is that the execs who could make those decisions are convinced that LLM tools obviate those needs, and shift all spare resources in that direction. But without heavy constraints from experienced humans, LLM tools will only multiply the complexity at the root cause of these sorts of outages.

That would be a good idea, until you realize that everyone you send emails to (in a business context) also uses MS 365.

Big tech are trying to take this option away from us.

Monopolies can provide a bad service and not lose customers.

Instead I'm wondering if someone at OpenAI and Anthropic had a look to check what their agents were up to, just to be sure.

lol surely

many universities bought into the cloud offering back when it was still live@edu, shown as an upgrade path for existing on-premise dovecot/exchange systems. back then a student would be lucky to get more than 100mb for their mailbox, and maybe a gig of personal file storage (which was only accessible on campus or via a vpn, the latter many would not have access to). microsoft just offered a _better deal_ at the time

frog boiling means that many never noticed how big (and therefore unspecialised) microsoft was until it was too deeply integrated

It's much better UX if everyone within the company uses the same office suite/email, it makes collaboration much nicer when everything is well integrated. And people get used to one system.

My guess is that having duplicate system would cost (in money for double subscriptions and lost productivity) much more than the benefit of being available 0.01% more. Especially for universities where almost nothing is really urgent and can't be delayed by a day.

Because it's not DNS related.. you can still resolve it.. and sometimes reach the frontend- but they can't route the requests to email servers because they are being "overloaded"

Because MSFT is explaining it as a rogue process overrunning servers after start up.

The amount of racist memes targeting Indians on the replies.

In their defense it was probably insanely fragile prior to the AI assisted coding boom.

Given the nesting of services at MS, the fractious org structure, and the length of service of some of these systems it was already a bit of a tinder-house.

AI just provided the accelerant, and while I don’t work at MS I can imagine they’ve lost some serious tenured talent over the last 5 years. I’d be surprised if anyone is left that actually understands how it’s all glued together.

Or rather, that might be the case before this outage. I suspect the org has relearned a fair bit over the last day.

No one knows if this is AI related, but it's easy to imagine that where changes written by AI fail in production, if AI can't find the solution (which will be common as AI access to production systems is typically reduced), then the devs won't either, or at least it will take them some time.

Ugh. My sarcasm detector is down. Shouldn't have hosted on MS servers.

I do actually believe that LLMs and process around them will continue get better and become the accepted, but that there's going to be a 'fossil record' of a lot of problems having been seeded in these early years of AI that will take some time repair.

What a fancy way to say, "we highkey have no idea why it's broken"

Its microsoft 360 not 365 (days in year). So they have four more days.

Alas, Teams is working, although parts of it which are integrated from other services are not, like calendar (exchange), or file sharing (share point).

TIL that Yammer turned into that

Regular communication is an important part of outage/crisis communication, even if that communication is "nothing new to share".

It's actually nice to have some updates on what they're doing that go further than just "we're investigating."

Well then your one basket is your machine, or your backup system. And Office and Onedrive still work IIUC, so you're not really comparing apples to apples.

If only this year were a leap year Microsoft would be in the clear

Email is actually designed with intermittent connectivity in mind. The sending server buffers outgoing emails until they're accepted by the receiving server. Typical settings are to retry every 4 hours, with a permanent failure being counted after 3-7 days.

I would take 99.0pct self hosted over having to use outlook!