We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.

This is the inevitable consequence.

When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.

I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.

My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.

The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.

Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.

In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.

And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.

Exchange Online and Microsoft 365 in general doesn't run on Azure.

How did that AWS DNS incident go recently? Unplanned outages are unfortunate but common, and hubris is the problem when everyone assumes that they (or their pet vendor) are immune.

From this we can tell where Jim Roskind works.

To be fair to Microsoft (and I'm having trouble believing I just wrote that), the vast majority of everybody's outages are "self owns". If you don't change anything, you're not going to break anything, and major external events (fires, cable cuts etc) are rare and can usually be worked around.

Isn't a complex computer system, at the lowest levels, an example of a Rube Goldberg machine ?

If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.

In my experience it is always the "business" people that do not want to take the time to do it correctly.

> building Rube Goldberg machines that cannot be reasoned about

What a great phrase!, can I use that in conversation?