I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is lost or if anything is recoverable or when AWS UAE will become functional again so we can recover that region.

Finally, I have a very good argument for multi-region deployments ;))

that's my go to website atm: https://health.aws.amazon.com/health/status

▲

richsouth 2 hours ago | parent | next [-]

What do you mean 'finally' - surely 'redundancy' or 'natural disaster' is reason enough.

▲

RamblingCTO 32 minutes ago | parent | next [-]

To a lot of managers/startup execs this is something like "we won't ever need this". And I'd agree to some extent for not so important/rebuildable services. Depends on what you need. In startups, you don't have infinite time you can spend on stuff. But this makes a good case: if a geographical region only has one AWS region, don't keep data or run services that can't be easily rebuild somewhere else. In europe you can just pick two AWS regions and you stay in the same regions, UAE not so much.

▲

doubled112 34 minutes ago | parent | prev [-]

I don't think this changes anything. It is always the same argument for me.

"How often do those happen?"

	▲	RamblingCTO 29 minutes ago \| parent [-]
		That was my point: now we have a very specific case we can argue. I always used the "what if a plane crashes into the data center" argument. Now I can say: in one of my engagements, we lost a datacenter completely because of a drone attack. That's a first for AWS as well, but we can draw from reality, not hypotheticals.

▲

crossroadsguy 3 hours ago | parent | prev | next [-]

Severity: Disrupted

So if data won't be recoverable you all will mark it something like "Status: FUBAR" or some equivalent term?

▲

BonoboIO 2 hours ago | parent | next [-]

FUBAR, I would vote for that

▲

lazide 2 hours ago | parent | prev [-]

I wonder if ‘Apocalypse’ or ‘Molten Slag’ would be considered professional enough.

	▲	bayindirh 2 hours ago \| parent [-]
		I'd accept "X_X" as status.

▲

Hamuko 3 hours ago | parent | prev | next [-]

We didn't do multi-region deployments, but we did store database backups in a separate region just in case something really bad happened and our AWS region became unavailable. Also had a plan/some ready Terraform stuff in order to start setting up a deployment if it became apparent that the region wasn't coming back anytime soon.

IMO, if you're using AWS and not replicating your data somewhere else, this should be an eye-opener for you.

	▲	RamblingCTO 32 minutes ago \| parent [-]
		Not sure why everyone read this as me doing anything here, I'm a fractional CTO, which is kind of an advisor. Nothing invaluable will be lost tho. It's not the core platform, just a localized version for specific customers in the region.

▲

SirFatty an hour ago | parent | prev | next [-]

You don't already have a DR plan in place?

	▲	RamblingCTO 28 minutes ago \| parent [-]
		Of course, no worries. Nothing irreplaceable will be lost ;) It was meant as a general example to be used in future arguments for everybody in tech leadership.

▲

chinathrow 3 hours ago | parent | prev [-]

Any backups?

	▲	TacticalCoder 2 hours ago \| parent [-]
		> Any backups? Yup something has to be said too about good old offline backups on fat tapes.