I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."

I do think his choice of words was a bit suboptimal. But, it got the point across.

> Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".

Something about this feels wrong:

- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.

- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?

Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?

The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.

One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.

You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.

My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.

I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).

I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.

But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.

I think the problem is when the execs aren’t looking at change-for-the-better. Instead they’re looking to appoint blame. Moving to a position of assumed competency in your staff and taking the perspective of how-to-improve-next-time is a real step forward in maturity. My guess is it’s rare and blame is easier: most management doesn’t reach this level, instead it’s immature.

[delayed]

[delayed]

This article may be more appropriate for LinkedIn.

Interesting. At our org we always do both. Root cause, what are we changing. I assumed that was standard practice.

If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?

The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?

I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.

What they needed was improved pattern recognition vs if this / then that which comes from post mortems.

Every post-mortem I've done had:

1. timeline of events

2. impact

3. 5 whys — the details of how and why things happend

4. actionable items

I've never seen a post-mortem without actionable items.

I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.

It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.

Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.

Consider one of the problems listed in the article

>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".

The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.

It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.

The executive did not display empathy.

People need to tell their stories. They need to be heard and have their work and its difficulty respected.

That's what went wrong. Everything else is an abuse victim rationalizing their abuse.

If I recall correctly, in The Pragmatic Engineer, this was described simply as "provide options, not excuses" and is a critical way of building credibility.

This is similar to the approach in healthcare, at least my experience in pharmacy.

When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.

Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).

No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.

An SVP of engineering should care about the details, this is the essence of leadership. When you don’t care about the details you are just a manager and worthless i.e. you should be replaced with a leader.

This is a really really smart SVP. The answer to “how do we prevent this in the future” is to identify decisions that reduce that problem from happening again. This doesn’t have to be perfect. You can say “we are going to do X next time” and also say “but we don’t know how far X will work”. When the failure happens again, you can retire process X and move on to Y. The aim is to make a series of decisions that eventually get to the heart of the issue.

Everything else is a conversation that takes up space on a post-mortem or runbook.

What if you work at an organisation that doesn't ask "How do we prevent this from happening again?"

Love this idea; won't go over well at all with the other accountants I work with. MY CFO doesn't like graphs because they "hide too many details' and insists on everything presented to her being in a table. MY VP pulls out a calculator and re-does the math on tables in decks by hand to help him 'understand'.

A lot of leaders like to hide in the details because the bigger picture is a harder problem.

A Senior Vice President of ENGINEERING doesn't want the technical details?

My father who was also involved in IT tells a similar story where the executive cut him off and said "I asked you for the baby, not the labor pains."

We call this incident postmortem, part of a company process. After serious problems and firefighting, when the patches and hacks and people working 24 hour and it admins wake on a Sunday morning to do work, we sit down and try to figure out how this will _never_ happen again.

If you don’t know what happened how would you be able to tell if the changes are addressing the problem? If you don’t care about the problem being addressed why jump in a call in the first place?

Learned this doing incident reviews. If the exec asks for details it usually means they don't believe you yet, so "skip to what changes" is actually the good outcome.

> ...I'll empathise with you. Then it'll happen again.

I'm not so sure this means "I trust you."

> At first I thought "I don't want the details" sounded dismissive.

Well that is because it was delivered by someone who was actually being dismissive!

If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.

Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix.

But otherwise be very wary of this pattern of communication where "we" might mean "you". Anyone with any sense of risk management should know not to do this.

If you have not seen it, watch Margin Call. When you get to the boardroom scene [0] you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 feet level.

Beyond the drama of the way the scene ends, it's actually a very good model of how that interaction should play out for everyone — making it clear that everyone in the room shares in the "we" of it, including the decisionmaker.

I have been in rooms where it played out at least something like that, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.

And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.

[0] said scene, but really, really, watch the film if you never have:

https://www.americanrhetoric.com/MovieSpeeches/moviespeechma...

Thats funny. When I try to only give the conclusion, I'm often asked for the details.

If I'm to explain what needs to change, that will by definition address processes or lack thereof that led to the problem, no? Which is then explaining what happened and why in an indirect manner, so many of the proposed changes may not make sense without more context.

Why are these SVPs even needed anymore? They seem to do nothing of value.

If you ask them, of course they will make up a bs story with their fake answer. The fact is that they fail the ablation test.

I would not have said that. I would have asked for the details and tried to find the gaps because, almost always, something could have been done. Then the question is how much risk reduction are you willing to pay for.

The quote box at the start of the article is the essence of it, the rest is AI-speak dressing it up.

TLDR:

Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:

“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.

Then it'll happen again.

So I don't want the details. I want to know what we're changing.”