Remix.run Logo
fidotron 2 hours ago

> I have never seen an Ops team being rewarded for avoiding incidents

That's why their salaries are so high.

denysvitali 2 hours ago | parent | next [-]

Depending on the tech debt, the ops team might just be in "survival mode" and not have the time to fix every single issue.

In this particular case, they seem to be doing two things: - Phasing out the old proxy (Lua based) which is replaced by FL2 (Rust based, the one that caused the previous incident) - Reacting to an actively exploited vulnerability in React by deploying WAF rules - and they're doing them in a relatively careful way (test rules) to avoid fuckups, which caused this unknown state, which triggered the issue

fidotron 2 hours ago | parent [-]

They deliberately ignored an internal tool that started erroring out at the given deployment and rolled it out anyway without further investigation.

That's not deserving of sympathy.

esseph 2 hours ago | parent | prev | next [-]

Ops salaries are high??? Where?!?!

hnthrowaway0328 an hour ago | parent [-]

Definitely commands better salaries than us pitty DEs.

agoodusername63 19 minutes ago | parent | prev [-]

news to me.