This needs to be amplified. A lot of the updates carry all the hallmarks of leadership consistently denying the engineering teams the time to build out sufficient introspective and troubleshooting tooling, on the finance-supported rationale that it is not financially justifiable for such low probability events regardless of impact. And this can be as simple as allowing for enough shadow non-Production environments real-time twinning inputs at scale enough to reproduce thundering herd issues, for example.

Take 8:52 PM UTC, for instance. "While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly." So many questions flow from this.

They jumped from "affected servers have become unresponsive" straight to "restarting these servers". No granular troubleshooting of a random sample of the unresponsive cohort to identify the precise origin of the failure mode. Possibly their own organizational chaos prevents the team at the low granularity layer seeing the pattern of whole groups of servers, from effectively interacting with the right team to troubleshoot at the individual server, high granularity layer. I've seen that at many of my clients; sometimes the different teams at different granularity layers don't even have established engagement channels for when things blow up across those granularity layers. I could go on, but you get the drift.

If you obsess over internal tooling, you waste time yak shaving and ship nothing; if you only care about shipping features, technical debt accumulates until a major outage hits. Because cutting corners for speed is the path of least resistance, paying down that complexity later is much harder than maintaining discipline upfront.

The fact that they don’t report certain steps in their public communications during an outage does not mean those steps weren’t done.

Everyone wants to blame the bankers and the bean counters. As I get older I realize that businesses will happily forgo insane sums in the name of predictability and continuity and that it's the team leads and the middle managers who don't want the troubleshooting and the control panels because those would provide an quasi-audit log that would more readily quantify bad management. They see these systems as a threat, a way for information of what's really happening to leak around their control.

Of course not all team leads and middle managers are like this but too many of them are and the bad ones are a great fit for exploiting BigCo environments so BigCos act like a catch basin for them.