5 Comments

Summary:

Some post-storm lessons are no brainers — make sure you have the right fuel hoses as well as the fuel itself? Duh. Others may come as surprise.

Getty Images
photo: Getty Images

Just over a year ago, superstorm Sandy slammed into the New York metro area, destroying thousands of homes and wreaking havoc with the IT infrastructure in southern Manhattan and low-lying areas of New York and New Jersey. It was the costliest and most destructive storm of the 2012 hurricane season and by some accounts the second or third most expensive in U.S. history.

So what have we learned when it comes to safeguarding IT infrastructure?

Quite a bit: at least according to data center pros who weathered the storm and have some ideas for mitigating damage from similar events in the future, which given what we know about rising water levels, are bound to happen again.

Getty Images

Getty Images

1: Focus on the people, stupid

Big data-center providers have contingency plans to cover acts of nature or man, including provisioning food, water, and cots for on-site personnel. But until Sandy, most of these were designed for a few days of staffing. That changed with Sandy, said Raouf Abdel, regional operating chief for the Americas for Equinix.

“We will prepare for longer intervals. Before we’d stock up for a day or two maybe three, but Sandy showed use we need to go longer,” he said. “We found we needed better sleeping quarters, more water, more food, more coffee.”

Before Sandy, nobody seemed to imagine that highways, tunnels and subways could be out for days on end. Now there have to be plans in place for how personnel can get to the affected area, and for how other personnel can work remotely as effectively as possible.

2: Assess the state of facilities — whether or not you own them

Most data center providers know they need to build new facilities on high ground — Equinix sites in Secaucus and North Bergen, N.J. are above the 100-year flood plain, for example. But when it comes to weather-proofing, more is more.

Lobby at Verizon office at 140 West Street, New York post-Sandy

Lobby at Verizon office at 140 West Street, New York post-Sandy

Providers using third-party facilities need to keep a sharper eye on the physical security and stability of those sites. Ensuring that there are berms or other physical barriers to low-lying doors is a no brainer. Equinix has added waterproof steel doors to prepare for “several more feet of water,” Abdel said.

It’s natural to focus on basements when you think of flooding, but a flat roof can be almost as big an issue. Installation of rooftop gear and piping often leaves perforations that can lead to problems in high wind and rain. So, when you assess your physical plant, look up as well as down, said Mark Thiele, executive vice president of data centers for Switch, the Las Vegas-based SuperNAP.

Make sure there are updated blueprints and other building documentation on site and available to emergency personnel. In an emergency, they will need them.

3: Carefully assess the backup power situation

Even if you have plenty of fuel for backup generators, it won’t help if the generators themselves or the pumps to supply them get flooded in a basement. If this gear must stay on lower floors, make sure it’s fully encapsulated and waterproofed, said Michael Levy, analyst at 451 Research.

Post Sandy, service providers should also make sure they have roll-up generators as well as fuel hoses onsite and easily accessible, said Ryan Murphey VP of operations for PEER 1 Hosting. Oh, and make sure those hoses fit both the generators and the fuel trucks.

And, not to belabor the obvious, make sure there are lots of batteries, flashlights and headlamps stocked, operable and accessible.

4: Bone up on DIY weather prediction

While most data center pros keep an eye on big storms, Sungard Availability Services is making crowdsourced weather watching a part of its standard operating procedure.

As Sandy developed, Sungard staff watched various weather sites and found the amount and quality of information available to be invaluable. For example, the National Oceanographic and Atmospheric Agency sites that simulate storm conditions are “unbelievable,” said Nick Magliato, Sungard AS Chief Operations Officer. “If you map NOAA data to tide charts, you can see how a storm surge might manifest say down the Hackensack River,” he said.

A combo of NOAA, Google Earth and government topological maps, can show a lot about what a storm surge from a 100-year storm will look like in advance, he said. Instead of managing the generic risk posed by a hurricane hitting the Eastern Seaboard, you can simulate what a four-foot tidal surge would mean to southern Manhattan and low-lying New Jersey.  “And we don’t have to hire an environmental engineering firm to do it,” Magliato said.

5: Choose new locations wisely

Obviously in looking for new facilities, it behooves you to make sure they’re on high ground, away from the coastal areas impacted by Sandy and before that Hurricane Irene in August 2011. Two “100-year storms” in two years prompted lots of conversations around the wisdom of keeping data center facilities in surge-prone areas.

Omaha, Nebraska-based Scott Data Center has fielded lots of inquiries from New York metro companies in the past few years, said company president Kenneth Moreano. Major League Baseball, for example, has put its second “hub” at Scott Data Center. MLB’s primary hub — which serves much of its streaming media — remains in New York, but MLB is replicating that capability in Omaha. “Originally Omaha was going to be their disaster recovery or backup location but now we will be their first hub outside of New York City,” Moreano said.

At least “one data point” in most of these discussions with New York area companies is talk of storm-related outages, those infamous “bucket brigades of fuel,” he said. While financial services firms that need extremely low-latency connections to trading floors will keep assets close to Manhattan, there’s no compelling reason for the bulk of data center infrastructure to be in the immediate area given the risks — and given New York’s extremely high real estate prices.

Just some things to think about for next time. Gulp.

  1. This is all good for protecting a single data center as the owner, but as a customer the first step should really be getting another facility in a different geographical region. The ability to fail over to a different data center with no customer impact is the only way to truly protect against an entire region going down.

    It’s a lot easier (but still not easy) to do this now with the big providers because their offerings are standardised across data centers. Amazon has mostly the same services in each facility and it can be entirely automated using their APIs. We do this with our hosting provider, Softlayer, too. Depending on the failure scenario we can swap traffic from our primary in Washington to secondary in San Jose with no customer downtime in most cases, or with just a few minutes of downtime if the entire facility fails unexpectedly suddenly.

    Share
    1. agreed david… i am working on a follow up about NYC metro-area companies who have moved or plan to move more of their it capability far away from the coast.

      Share
  2. Another important point is to ensure you have a full understanding of the dependencies between your data centers, infrastructures, and applications. Even for many Tier 1 applications, there are critical dependencies both within the specific data center impacted and other data centers. A lot of folks found out the hard way with Sandy that they could recover their primary apps to back up sites, but hadn’t well documented all the other integrations necessary to stitch their business processes together across those applications. That meant that it took longer to get the business back up and running than it should have if this tribal knowledge had been well documented.

    Share
    1. Another great point. thanks for the comment @mselheimer

      Share
  3. Fewer local datacenters. More managed and/or public cloud services. Rather than just “choose new locations wisely”, we should also choose new partners the same.

    Share

Comments have been disabled for this post