I hope everyone involved in working on this is staying grounded and relaxed and not suffering a ton of physical stress over what is ultimately the result of greedy leadership decisions :(

▲

cucumber3732842 2 days ago | parent | next [-]

Considering the middle management culture at MSFT and other tech BigCo's it's probably better that this be a huge public outage where their crappy managers will actually face some heat for creating the kinds of processes that lead to these outcomes rather than can throw the team under the bus when inevitable human error creeps into their obviously un-followable at scale over time process.

▲

franktankbank 2 days ago | parent | prev | next [-]

I hope they all quit cause microsoft fuckin sucks.

▲

NetMageSCW 2 days ago | parent [-]

Better than most alternatives.

▲

Ygg2 2 days ago | parent [-]

Say what you want about OpenOffice it doesn't experience 24h degradation because some idiot pushed vibe slop to production.

▲

EvanAnderson a day ago | parent [-]

Not to defend Microsoft, but the thick-client Office applications (Word, Excel, PowerPoint, etc) worked fine during the outage. Nobody couldn't use Excel because of this outage. Outlook worked fine too if you were connecting it to an on-premises Exchange instance.

This was an outage of a server-hosted database for email, not an outage of desktop application software.

	▲	Ygg2 a day ago \| parent [-]
		Sure. Just wait till it gets fully entshittified. With thin clients you can enjoy any MSFT fuckup as free time off!

▲

yourapostasy 2 days ago | parent | prev [-]

This needs to be amplified. A lot of the updates carry all the hallmarks of leadership consistently denying the engineering teams the time to build out sufficient introspective and troubleshooting tooling, on the finance-supported rationale that it is not financially justifiable for such low probability events regardless of impact. And this can be as simple as allowing for enough shadow non-Production environments real-time twinning inputs at scale enough to reproduce thundering herd issues, for example.

Take 8:52 PM UTC, for instance. "While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly." So many questions flow from this.

They jumped from "affected servers have become unresponsive" straight to "restarting these servers". No granular troubleshooting of a random sample of the unresponsive cohort to identify the precise origin of the failure mode. Possibly their own organizational chaos prevents the team at the low granularity layer seeing the pattern of whole groups of servers, from effectively interacting with the right team to troubleshoot at the individual server, high granularity layer. I've seen that at many of my clients; sometimes the different teams at different granularity layers don't even have established engagement channels for when things blow up across those granularity layers. I could go on, but you get the drift.

If you obsess over internal tooling, you waste time yak shaving and ship nothing; if you only care about shipping features, technical debt accumulates until a major outage hits. Because cutting corners for speed is the path of least resistance, paying down that complexity later is much harder than maintaining discipline upfront.

▲

NetMageSCW a day ago | parent | next [-]

The fact that they don’t report certain steps in their public communications during an outage does not mean those steps weren’t done.

	▲	yourapostasy a day ago \| parent [-]
		Fair point. Maybe at their scale they’ve learned something different about crisis management than I’ve experienced on a smaller scale. Would love to hear from those who have been in the trenches at a similar scale as these past couple of days. In a 1-2 hour unplanned outage of typically way smaller systems at some of my clients, I’m generally able to leave my status reports at this level we see here. But as the time and blast radius expands, I find it builds more trust that we’re doing everything within our power and builds more reassurance by sharing progressively more detail. The increasing detail also unlocks massive offers of assistance from many areas of the organization in many cases. Telling stakeholders what my team and I have tried and ruled out often helps them explain to their stakeholders just as much as telling them what has worked.

▲

cucumber3732842 2 days ago | parent | prev [-]

Everyone wants to blame the bankers and the bean counters. As I get older I realize that businesses will happily forgo insane sums in the name of predictability and continuity and that it's the team leads and the middle managers who don't want the troubleshooting and the control panels because those would provide an quasi-audit log that would more readily quantify bad management. They see these systems as a threat, a way for information of what's really happening to leak around their control.

Of course not all team leads and middle managers are like this but too many of them are and the bad ones are a great fit for exploiting BigCo environments so BigCos act like a catch basin for them.