Remix.run Logo
▲ kccqzy 4 hours ago

I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.

> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.

Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?

SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.

▲motionlessveloc 2 hours ago | parent [-]

I'm glad you apparently live in an ideal world where every incident is fully root caused and fixed at the origin, and all your coders produce perfect, bug-free code.

Most of us, however, do not.

▲kccqzy 38 minutes ago | parent [-]

Every incident having a root cause is required. If your organization doesn’t do this, it basically means they don’t value reliability at all. The best SREs quit. Well either that or the SREs have the ability to transfer oncall responsibility to SWEs. Whether or not that can be fixed at the origin is an entirely different matter. Bug-free code doesn’t really matter: the best incidents are the kind that reveals hitherto unknown bugs.