Here's a theory: A network change made connection setup more expensive or forced mass reconnects, saturating the front ends’ secure-connection process. Saturated servers fail health checks and get pulled, so the survivors take more load and fail too. Every client that gets dropped reconnects, and each reconnect needs a fresh secure handshake. That is why restarted servers die as soon as traffic returns. The outage now sustains itself even with the trigger being reverted, hence the traffic blocking and staged reintroduction. The most recent surge is likely retry amplification, partly caused by their own traffic rebalancing.

▲

mixdup 2 days ago | parent | next [-]

They should be able to mitigate that at the border. Whatever load balancing solution is in place should just drop connections before they get to a point that can overload the system. That they haven't, or can't, makes me think it's more complicated than "it's hard to bring it back up when everyone is trying to get back in"

▲

cheeze 2 days ago | parent | next [-]

Agree. For a large cloud provider, the thundering herd retry-storm is pretty standard failure scenario and a "greatest hit" outage at this point.

Throttle the hell out of everything, then slowly ramp traffic back up.

I'm confident they've already tried this exact thing. This is cloud 101 in 2026.

	▲	Eji1700 2 days ago \| parent [-]
		So i know fuck all and am basically just rubbernecking the car crash at this point, but I would've agreed however they just dropped this gem: > Current status: As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability. This includes traffic optimization actions such as traffic throttling, protocol adjustments, and failover routing intended to improve stability in active regions. In parallel, were expanding internal testing to track activity within a contained environment to improve traffic monitoring and issue detection capabilities. That...kinda reads like they're just trying it? I'm assuming there's something being missed or it's a "fuck it try it again" but I do wonder if some story is going to come out of people in the decision making seats who don't know what they're doing.

▲

franktankbank 2 days ago | parent | prev [-]

Could be a bunch of dumb dumbs.

▲

srinidhin 2 days ago | parent | prev | next [-]

Also called meta-stable failures - https://muratbuffalo.blogspot.com/2023/09/metastable-failure...

	▲	contravariant 2 days ago \| parent [-]
		Sounds like it running correctly was the meta-stable state. Failure is actually more stable since it can reach a failed state from the ground state of simply not running.

▲

tguedes a day ago | parent | prev [-]

Isn’t this very similar cause to the AWS us-east-1 outage from last year. I thought I remember it being about a network change that overloaded servers, leading to a cascading failure.