I have never seen an Ops team being rewarded for avoiding incidents (focusing in tech debt reduction), but instead they get the opposite - blamed when things go wrong.

I think it's human nature (it's hard to realize something is going well until it breaks), but still has a very negative psychological effect. I can barely imagine the stress the team is going through right now.

▲

fidotron 2 hours ago | parent [-]

> I have never seen an Ops team being rewarded for avoiding incidents

That's why their salaries are so high.

▲

denysvitali 2 hours ago | parent | next [-]

Depending on the tech debt, the ops team might just be in "survival mode" and not have the time to fix every single issue.

In this particular case, they seem to be doing two things: - Phasing out the old proxy (Lua based) which is replaced by FL2 (Rust based, the one that caused the previous incident) - Reacting to an actively exploited vulnerability in React by deploying WAF rules - and they're doing them in a relatively careful way (test rules) to avoid fuckups, which caused this unknown state, which triggered the issue

	▲	fidotron 2 hours ago \| parent [-]
		They deliberately ignored an internal tool that started erroring out at the given deployment and rolled it out anyway without further investigation. That's not deserving of sympathy.

▲

esseph 2 hours ago | parent | prev | next [-]

Ops salaries are high??? Where?!?!

	▲	hnthrowaway0328 an hour ago \| parent [-]
		Definitely commands better salaries than us pitty DEs.

▲

agoodusername63 16 minutes ago | parent | prev [-]

news to me.