Remix.run Logo
abeppu 3 days ago

I wasn't involved in the specific details but I remember being told that during the power outage from hurricane Sandy, even datacenters that had sufficient generators had trouble getting the diesel to keep them running, because everyone wanted diesel at the same time and both the supply and distribution were bottlenecked.

How long can most DCs run with just the fuel onhand? Have standards around that changed over time?

jabart 3 days ago | parent | next [-]

At a call center, that had a whole datacenter in the basement. They had two weeks of fuel on hand at all times. Being on a border of a state, they also had a 2nd grid connection in case one failed.

The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.

Bender 2 days ago | parent | prev | next [-]

How long can most DCs run with just the fuel onhand?

There really is not a universal answer for this. Every generator will have what is called a "day tank" that as you might guess lasts for one day under a nominal load.

The day tanks are connected in pods to large diesel fuel tanks. Every {n} number of generators get a main tank. Those tanks vary in size depending on how much the company wishes to spend and how resilient they need to make their data-center excluding fuel trucks. Cities have regulations about how much fuel can be above or below ground at each location. My main tanks were 10K gallons. Each generator used over a gallon per minute under load.

And you are right, during a regional or global disaster fuel trucks will be limited. They who bribe the most get the last fuel but that too will run out. Ramping up production and distribution takes weeks and that assumes roads are still viable and the internet outside of the data-center is still functional.

stevetron 3 days ago | parent | prev | next [-]

It seems to me that Hurricane Sandy caused an issue with fueling backup generators in a New York City datacenter, where there was a bucket-brigade of people carrying fuel in pails up flights of steps several stories.

abeppu 2 days ago | parent [-]

I remember hearing this story at the time but have forgotten all the details. I think it involved a pretty well known company? I also remember hearing that some DC in NJ got special priority in getting diesel in the following days because some federal government services were hosted there and so it was treated as a national security issue to keep them supplied.

kalleboo 2 days ago | parent [-]

It reminds me of the blog during Hurricane Katrina, with the guy carrying barrels of fuel up the stairs to the 10th floor https://en.wikipedia.org/wiki/Interdictor_(blog)

stogot 3 days ago | parent | prev [-]

I recall ranges I’ve heard from operators as 24h to upwards of 72 (rare)