Remix.run Logo
sys42590 9 hours ago

It would be interesting to hear their contingency plan for any kind of disaster (most commonly a fire) that hits their data center.

sschueller 9 hours ago | parent | next [-]

Yep, does anyone remember the OVH fire[1][2]?

[1] https://www.techradar.com/news/remember-the-ovhcloud-data-ce...

[2] https://blocksandfiles.com/wp-content/uploads/2023/03/ovhclo...

otherme123 7 hours ago | parent | next [-]

I fully lost three small VPS there, and their response was poor: they didn't even refund time lost, they didn't compensate for time lost (e.g. a couple of months of free VPS), I got better updates from the news than from them (news were saying "almost total loss", while them were trying to convince me that I had the incredible bad luck that my three VPS were in the very small zone affected by the fire). The only way I had to recover what I lost was backups in local machines.

When someone point out how safe are cloud providers, as if they have multiple levels of redundancy and are fully protected against even an alien invasion, I remember the OVH fire.

wiether 3 hours ago | parent [-]

OVH VPS is not the same as say, AWS EC2.

It's their "Compute" under "Public Cloud" that is competing against AWS EC2. https://us.ovhcloud.com/public-cloud/compute/

They handled the fire terribly and after that they improved a bit, but an OVH VPS is just a VM running on a single piece of hardware. Quite not the same thing as the "Compute" which is running on clusters.

AndroTux 7 hours ago | parent | prev [-]

contingency plan: Don't build your data center out of wood.

srg0 7 hours ago | parent [-]

Plastic is made from the same stuff as gasoline.

direwolf20 5 hours ago | parent [-]

Drain cleaner and hydrochloric acid makes salt water. Water is made of highly explosive hydrogen. Salt is made of toxic chlorine and explosive sodium.

fpoling 8 hours ago | parent | prev | next [-]

They use the datasenter for model training, not to serve online users. Presumably even if it will be offline for a week or even a month it will not be a total disaster as long as they have, for example, offsite tape backups.

instagib 9 hours ago | parent | prev | next [-]

Flooding due to burst frozen pipe, false sprinkler trigger, or many others.

Something very similar happened at work. Water valve monitoring wasn’t up yet. Fire didn’t respond because reasons. Huge amount of water flooded over a 3 day weekend. Total loss.

twelvechairs 9 hours ago | parent | prev | next [-]

Theres only one solution to this problem and its 2 data centres in some way or form

mbreese 8 hours ago | parent [-]

What's the line from Contact?

why build one when you can have two at twice the price?

But, if you're building a datacenter for $5M, spending $10-15M for redundant datacenters (even with extra networking costs), would still be cheaper than their estimated $25M cloud costs.

golem14 8 hours ago | parent [-]

Or build two 2.5MM DCs (if can parallelize your workload well enough) and in case of disaster, you only lose capacity.

You need however plan for 1MM+ pa in OPEX because good SREs ain’t cheap (or hardware guys building and maintaining machines)

direwolf20 5 hours ago | parent | prev [-]

the plan is to not set it on fire. If your office burns down you are already screwed