We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.

This is the inevitable consequence.

When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.

I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.

My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.

The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.

Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.

In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.

And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.

Exchange Online and Microsoft 365 in general doesn't run on Azure.

How did that AWS DNS incident go recently? Unplanned outages are unfortunate but common, and hubris is the problem when everyone assumes that they (or their pet vendor) are immune.

From this we can tell where Jim Roskind works.

To be fair to Microsoft (and I'm having trouble believing I just wrote that), the vast majority of everybody's outages are "self owns". If you don't change anything, you're not going to break anything, and major external events (fires, cable cuts etc) are rare and can usually be worked around.

Isn't a complex computer system, at the lowest levels, an example of a Rube Goldberg machine ?

If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.

In my experience it is always the "business" people that do not want to take the time to do it correctly.

> building Rube Goldberg machines that cannot be reasoned about

What a great phrase!, can I use that in conversation?

They did look at recent changes but none seemed to be related, but it turned out recent network changes triggered a failure mode in a much older network change. Dependent or cascading issues remain hard to diagnose or find.

Could be something adjacent - the last time I majorly blew up a release was because something had changed (which was fine) but it caused a list of items to be broken over two pages, but only in prod. Me and a team mate were scratching our heads for the longest time because "what had changed" was quite mundane itself.

Azure is still humming along working fine.

I disagree; the communication aspects - including external - are a hugely important part of any incident response. IME the biggest challenges in these situations are the many levels & combinations of communication and coordination. You don't need your best DevOps person doing it, but there should be plenty of competent technical product managers or engineering managers who could do this well.

Microsoft is not a small startup where the CTO is getting pulled into firefighting.

They should have established procedures for large-scale events, and they must include a coordinator whose entire job is to manage the external and internal communication.

Individual talent is irrelevant when you have a company culture that rewards the opposite of that. Any talented person working at Microsoft knows that they are dealing with structural problem that force MS products to be massive piles of shit. There isn't much an talented individual can do about that.

Stuff happens and things break everywhere but right now at Microsoft they break badly and often, that points to actual issues.

Almost certainly not the fault of individual people, it's a management failure of course

i agree with everything you said here except the last part.

lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%

The enshitification of Windows was not technical incompetence; it was intentional product direction.

They transformed Windows from a useful OS to a banal surveillance and advertising platform.

Microsoft's superpower has always been hiring smart people and making them work under dumb management to create flimsy, mediocre outcomes.

I'm not actually sure this is true anymore. It seems most of their talent has left.

There are lots of talented amateurs. What separates professionals is standards, quality, teamwork, etc.

I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.

Don't worry, their replacements will receive very clear instructions instructing them to make no mistakes.