| ▲ | toast0 an hour ago | |
> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start. From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues). Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully). Incidentally, the best way to make sure load shedding works is to have to use it frequently! In my last job, the seemingly unchangeable client settings that resulted in load spikes every hour on the hour were super annoying, but also provided hourly confirmation that basic automatic load shedding worked. | ||