Remix.run Logo
▲ mike_hearn 9 hours ago

No big surprises here. Big outages at big firms seem to always look like this.

The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.

This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.

It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.

Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.

▲nilptr 2 hours ago | parent | next [-]

There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".

▲CobaltFire 2 hours ago | parent | prev | next [-]

I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.

We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.

Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.

▲TomatoCo an hour ago | parent [-]

I imagine the predefined list is easier when it's all consumers that are under your control. I figure with civilian power some random percent of houses'll kick their AC on after power is restored.

▲Rygian 2 hours ago | parent | prev | next [-]

Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.

It went remarkably well.

▲rawgabbit 4 hours ago | parent | prev | next [-]

It's more testing theater on their end; it's just perfunctory rubber stamping. There was and is no real performance testing under simulated near-real world conditions.

▲solyin 2 hours ago | parent [-]

Did you do anything to try and fix that?

▲rawgabbit an hour ago | parent [-]

Yes. As another commenter said, you can partition the clients into pods-clusters. And use your test suite software against a realistic pod-cluster. Azure has such a test suite: https://learn.microsoft.com/en-us/azure/app-testing/load-tes...

▲toast0 22 minutes ago | parent | prev | next [-]

> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster.

I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.

From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).

Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).

▲tonfa 3 hours ago | parent | prev | next [-]

Somewhat related: https://company.sbb.ch/en/railway-development/services-rus/e...

SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)

▲nickdothutton 2 hours ago | parent [-]

3-5 days in the UK according to "regulations". Some talk of it potentially taking 7-14 days under certain bad circumstances.

▲brunoborges 2 hours ago | parent | prev | next [-]

The biggest challenge of bringing a resilient system back online after an outage, is the resiliency of said system fighting back.

▲scheme271 2 hours ago | parent | prev | next [-]

Texas was on the verge of having to do a black start a few years ago when winter weather started causing failures. I think people were quoting potentially weeks to get everything back online. As it was, I think people didn't have power for days during sub-freezing temps due to load shedding.

▲ 3 hours ago | parent | prev | next [-]
[deleted]
▲2muchcoffeeman 2 hours ago | parent | prev | next [-]

Good to know even Google introduces circular dependencies!

▲jvidalv 3 hours ago | parent | prev | next [-]

Happened to Spain last year. The black start.

▲EGreg 7 hours ago | parent | prev | next [-]

In software servers this is called the “thundering herd” problem. It is why clients are coded do not just exponential backoff but random offsets too.

▲chinathrow 7 hours ago | parent | next [-]

It's also a well known problem, so no excuse for that.

▲dijit 3 hours ago | parent | next [-]

a well known problem with more than a few known fixes, one of which is load shedding your upstream based on protocol aware load balancrs.

I've had similar issues where backends couldn't come up due to load killing them as they came up: my solutions was to turn off the load balancers until the backends all came up before opening the door again...

I agree: no excuse.

▲ 3 hours ago | parent | prev | next [-]
[deleted]
▲VirusNewbie 2 hours ago | parent | prev [-]

Not only did they not handle thundering herd, it happened across all regions.

▲ an hour ago | parent | prev [-]
[deleted]
▲nialv7 2 hours ago | parent | prev | next [-]

that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.

▲avianlyric 2 hours ago | parent | next [-]

It’s both, but every grid has a sub-set of power stations with black-start capability (a crap ton of diesel generators), that exists to break the circular dependency problem and bootstrap the rest of the grid.

But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.

▲bragh 2 hours ago | parent | prev [-]

That was a very enlightening moment in Factorio when I had already converted everything to power and the whole grid blacked out and I needed to get it going somehow again.

▲xunil2ycom 4 hours ago | parent | prev [-]

Well stated.