I have lived through and debugged cascading failures and large scale distributed systems.

The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.

Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).

A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.

We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.

This is the inevitable consequence.

When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.

I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.

My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.

The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.

Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.

In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.

And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.