Remix.run Logo
hintymad 5 hours ago

> Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts.

Not to be sarcastic but just to offer an observation: in a sufficiently large or bureaucratic organization, preventing an incident from happening can rarely get you any credit or visibility. Such achievement falls into the bucket of "what you're supposed to do". So, those who navigate company dynamics well would rather let the incident happen and then be loud on the follow-up action items. The trick is not to turn an incident into a diaster, so it's a dedicate act.

tormeh 5 hours ago | parent | next [-]

I learned this early in a conservative org. Preventing things is risky. Just keep the solution ready for when things go wrong because then you'll get approval.

sergeym 3 hours ago | parent | next [-]

I've had this in my career also, I've had a solution that was deemed too risky to release by management but would have prevented an outage, but when an outage actually happened that was the first thing they wanted to try and it worked gloriously. I'm thinking that if it was released prior to the incident it would have not have had the same impact on my career.

justonceokay 2 hours ago | parent | prev | next [-]

Anecdotally I know of an engineer in the Excel team. They would keep around a list of low priority security bugs. When they wanted to do improvements on the system they would attach it to one of the security bugs “nearby“ because the change would become approved much more easily than just fixing the problem itself.

cloche 4 hours ago | parent | prev [-]

Never waste the opportunity a good crisis presents

nitwit005 4 hours ago | parent | prev | next [-]

The examples of high impact all seem like things unlikely to receive recognition.

If you save a sales deal, they'll cheer the sales staff. And pay them a commission, which you will receive no part of.

skmurphy 4 hours ago | parent [-]

And start to build a relationship with sales that, at least in a B2B firm, can be of significant benefit.

CobrastanJorji 4 hours ago | parent | prev | next [-]

Also, disaster is a good signal to the higher ups that there are problems in your org. If you keep putting out every fire with heroics, your boss will know (maybe), but his boss's boss's boss will see that your org is doing great and everything is all code green.

If you let a few things burn down, your boss's boss's boss will notice the fire, and things may improve. It's perhaps the easiest way you have of communicating with them.

hintymad 3 hours ago | parent | next [-]

Yeah, communicating upwards about potential issues and what you're doing about them is essential. When a disaster strikes, you'll look like a sage in the eyes of the higher ups.

themaninthedark 3 hours ago | parent | prev [-]

What if my boss's boss is the root cause of the fires and he just blames the "team" if his boss ever sees a hint the of issues?

CobrastanJorji 33 minutes ago | parent [-]

Your great grandboss will not look favorably upon your grandboss passing blame downhill. Disasters are (hopefully) always the fault of the person in charge, which is why you'll notice that your bosses are unusually involved with meetings about how to avoid more disasters.

mjklin 3 hours ago | parent | prev | next [-]

It’s about what people notice. In town governments I’ve heard of cutting popular programs that will provoke an outcry only to get credit for reinstating them, while possibly smuggling through other actions that are necessary but unpopular.

johnnycool 3 hours ago | parent | prev | next [-]

Many carears are made and bonuses paidout with heroesim trick.

bauldursdev 3 hours ago | parent | prev [-]

Can't solve a problem that doesn't exist yet