    We are experiencing an issue with multiple services in
    europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
    
    Due to a power event we had suffered an outage within
    europe-west2-b zone. The power event is mitigated. Partial
    mitigation to infrastructure services will be observed within the
    next 30 minutes and the remaining services will be mitigated there
    after.

So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.

(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)

▲ bsuvc 2 hours ago | parent | next [-]

I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.

For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...

▲ td521534f12 9 minutes ago | parent | prev | next [-]

GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.

▲ robinson-wall 2 hours ago | parent | prev | next [-]

> I'm amazed the big companies reported as down weren't using multiple zones

Much as with backups - if you don't test them, they don't exist.

Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.

	▲	forgotaccount3 36 minutes ago \| parent [-]
		We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands. It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time. We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.

▲ crumby an hour ago | parent | prev | next [-]

Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.

	▲	chrisjj 19 minutes ago \| parent [-]
		The drawback being temptation not to power them indepedently, it seems.

▲ HelloUsername an hour ago | parent | prev | next [-]

> https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...

https://news.ycombinator.com/item?id=50163858

▲ cavoirom an hour ago | parent | prev [-]

I know some banks out-sourced most of its infrastructure to external operators, and some operator I worked with didn't seem competent.