3 Comments

Summary:

Data centers up and down the eastern seaboard went into emergency mode to keep up and running as Hurricane Sandy churned northwards. But the key to survivabilty is not last-minute prep but long-running practices.

Terremark data center
photo: Terremark

Updated: As Hurricane Sandy swept north and toward landfall, data centers from Florida to Boston and further north did their things to prepare.

Verizon’s Terremark unit went on high alert last week as Sandy approached Terremark’s Miami-area sites. Then it replicated the process for its major facilities in Culpepper, Va., and the New York area, said spokesman Xavier Gonzalez.

Sandy is bringing wind gusts of up to 70 miles per hour in New Jersey, and the related storm surge could reach record levels of between 6 and 11 feet.

“From our perspective, this is part of doing business — we build data centers with these situations in mind. And from the cloud computing perspective, people have to remember that the cloud lives in data centers,” said Gonzalez. (Verizon is posting regular Sandy updates here.)

Update: Internap likewise started ticking off checklist items early, initiating its Emergency Action Plan for New York and Boston data centers late last week.

These plans — which are customized for each data center — call for testing out emergency systems, topping off fuel supplies and ensuring food and water availability for on-site personnel for the storm’s duration, said Bill Brown, VP of data center operations via email.

In addition:

“Each facility has enough fuel to run on generators for multiple days and we have already engaged our fuel suppliers for on-call assistance, if needed. We also added more engineers and technical support staff at each data center for the duration of the storm to monitor and ensure infrastructure availability and provide ‘remote hands service’ for customers so they do not have to travel to the data center.” 

As Data Center Knowledge pointed out, most of Manhattan’s data center sites are outside the lowest-lying Zone A which has already been evacuated — with the important exception of 75 Broad St.

Keeping an eye on Amazon US-East

Given several highly publicized Amazon outages over the past year, many will be watching Amazon Web Services’ US-East data center complex in Ashburn, Va. Last June, severe thunderstorms kicked off a chain reaction that led to a major outage there, ultimately impacting Netflix and other large AWS customers. Amazon US-1 suffered a non-weather-related event last week, as well.

Update: Heroku is keeping a watchful eye on AWS US-East, according to a post to its status page late Monday night.

Cloud storage provider Nirvanix said Monday it will enable customers using its New Jersey Node 4 data center to move their data to other locations in its cloud storage network — either temporarily or permanently — at no charge.

The key to data centers surviving a major natural event is not to wait until the last minute to prepare, said Shannon Snowden, a data center professional who is now senior technical marketing architect for Zerto, a company with technology that helps companies move and failover their mission critical applications.

Word to the wise: prepare, prepare, prepare

“You can’t wait ’til folks’ hair is on fire to plan these things,” he said. “What you should be doing from the data center perspective is [always] make sure the power has been tested, that you can fail over to generators, that those those generators are tested to make sure they’re functional and that they have enough fuel,” he said.

And, it’s critical to make sure that disaster recovery and secondary sites are able to ramp up when needed. “These places are typically sleepy little sites, quiet and dark most of the time with skeleton crews. You need a staffing plan to go from sleepy to all hands on deck,” he said.

The real worry with hurricane type events is not so much the wind, which any self-respecting data center can withstand, but flooding, which can cause huge problems. For that reason, drainage and landscaping are critical.

Hurricane Sandy photo courtesy of Flickr user NASA Goddard Photo and Video.

You’re subscribed! If you like, you can update your settings

  1. PaaS provider has their plan up for Sandy here: https://status.heroku.com/incidents/460

    Thanks for the updates Barb!!

  2. Anonymous Coward Friday, November 9, 2012

    Build your DCs in Alaska. There’s no hurricane there, and cooling is for free.

  3. Cloud Server Hosting Friday, December 7, 2012

    Hurricane Sandy has definitely taught the community a few lessons, but overall it has reinforced the notion that an effective, efficient disaster recovery plan is essential for every business that relies on stored data. Cloud hosting is a great option for disaster recovery because it is flexible, reliable, and cost-effective.

Comments have been disabled for this post