9 Comments

Summary:

Peer 1 Hosting’s small Manhattan-based staff, along with friends and customers, managed to keep the company’s generators at 75 Broad Street going with good-old fashioned brute labor. For two days they’ve hand-carried buckets of diesel fuel up 17 flights. Talk about above and beyond…

75 Broad Street in better times.
photo: Google Street View

Updated:When Hurricane Sandy hit 75 Broad Street in Manhattan, forcing Con Ed to cut power and then flooding the basement generators, the most customers could legitimately expect was an orderly shutdown of their equipment. But Peer 1 Hosting’s small local staff — along with friends and customers —  went way beyond the call of duty, forming a bucket brigade to deliver gallon upon gallon of diesel fuel up 17 floors to keep the company’s backup generator humming — and their equipment online.

The ongoing effort, flagged for me by Tier1 Research analyst Carl Brooks, is truly super human. “Seventeen flights? These guys don’t just deserve more customers, they deserve a freaking bonus,” Brooks said via email.

The work, which has been ongoing for more than two days now, is led by Mike Mazzei, the company’s head data center manager, and technician Scott Debernardo. They did their due diligence last week as the storm bore down. They arranged for food and water to be brought in and for fuel delivery. All that worked as planned. But, as in any disaster, there were wild cards.

For example, there’s a connection for the fuel trucks to fill the basement tanks, but there is not one to pump diesel up 17 flights of stairs to Peer 1′s other generator. That meant brute force had to be applied. A 55 gallon barrel was set up at the entry point, and another was left at the top of the stairs. The five local Peer 1 staffers, along with friends and customers, schlepped the fuel up by hand in buckets. The generators burn through 40 gallons every hour.

On Monday, there were anywhere from 10 to 15 people working at any given time and this morning there were 35, said Ryan Murphey, VP of data center operations who spoke to me from Peer 1′s San Antonio office.

Anthony Casalena, founder and CEO of Squarespace, a Peer 1 customer, went to the site to ensure a clean shutdown, found the bucket brigade, and ended up helping, as he writes here. As of SquareSpace’s latest update at 10:38 a.m. EDT Tuesday:

“Bucket brigade going strong. We’ve gone through half of our morning fuel delivery and are expecting a truck with 5,000 gallons coming at noon. Potential issues are lack of bucket brigade manpower into the night, and our fuel pumps burning out.”

Update (Nov. 1 at 9:29 a.m. EDT): Fogcreek Software, another customer, is also heavily involved with the diesel brigade effort as outined in its status page. According to Fogcreek fuel pumps are on their way as of Thursday morning but are stuck in gas lines in New Jersey.

also As Tier1′s Brooks pointed out, people who worry about moving workloads to the cloud or third-party provider should take this story to heart. “You should call this story ‘why your data actually is safer in the cloud — cuz your data center people ain’t doing this s**t,’ or ‘Yes, Virginia, you can trust your data center operator.’”

Here’s the other thing. As much as we virtualize and automate and build redundancy, computing — whether it’s “in the cloud” or in your server closet — still depends on such lower-tech things such as a power supply and, just as importantly, human ingenuity.

75 Broad Street photo courtesy of Google Street View.

  1. Reblogged this on The Perkins Journal and commented:

    This is so freaking cool.

    Share
  2. Lorenz Redlefsen Wednesday, October 31, 2012

    If they have less powerful pumps, they could put one at street level, and another halfway up (say, the 8th or 9th floor), so that less powerful pumps can do the job. What matters is the height of the fuel column the pump is supporting.

    Squarespace’s 12:40pm update says: “The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.”

    Share
    1. right i know that Peer1 and other tenants — inap etc were looking at pumping alternatives…bucket brigade is certainly not meant to be a long-term fix.

      Share
  3. Inspiring, but the reality is that they haven’t maintained uptime for many of their customers during this window despite these above-and-beyond efforts. Bad situation, but the fact remains that this a true “outage” at Peer1 for many companies (including mine.)

    Share
    1. Are you sure it wasn’t actually an outage upstream from PEER 1? There are many companies that connect into PEER 1 and vice versa that have had and continue to have outages, which Peer can’t help. I’m not really sure its credible to say they had power-related downtime or downtime that affected their customers that was their fault. This would be a little more informative if you shared some details about your experience.

      Share
      1. From http://forums.peer1.com/viewtopic.php?f=37&t=7532#wrap

        06:30 PT – We are going to implement a controlled shutdown of NY Data Center at 10:45 ET. Customer communications is being prepped.

        At this time an email went out to my company and I’m assuming most others at 10:25AM EST telling us we had 30 minutes to shut everything down:

        “PEER 1 Hosting¹s local facilities team has been on-site at our NYC data center, maintaining our systems throughout the night while running on generator power. Due to the severe impact of this storm on the infrastructure in Manhattan, we are requesting all our customers hosted in our NYC data center to plan on gracefully shutting down your equipment between now and 11:00am EST today.”

        I’m assuming hundreds of machines were powered down. Our company issued a controlled shutdown of all our servers there to avoid the impending abrupt power failure — including many that are mission critical to my business. The other option was potential data loss and/or hardware damage associated with an unplanned power loss / brownout / blackout.

        I can only assume that this dramatic decrease in load contributed to subsequent forum posts about how the fuel was lasting “much longer than expected.”

        Anyhow, we were down for quite some time, and still have many important machines we haven’t powered up yet. We turned on some machines after reading the hourly updates that “Generators are running and providing power.” Still, some mission critical data is dark — including a fairly large cluster — until we have confidence that the new fuel pump and AC issues have been worked out to the point that an unexpected blackout isn’t a likelihood.

        Share
  4. Lee Colleton ★ Wednesday, October 31, 2012

    “The Cloud” is tinged with diesel exhaust.

    Share
  5. Heather Singer MacKay Thursday, November 1, 2012

    What impresses me about this story is that this is an amazing example of a true customer service culture, and so much so that customers are participating in the business continuity! Even if there have been some outages, as alleged in other comments, at this level of disaster, this is inspiring. Good for Peer1 staff!

    Share
  6. Whatever they are doing beats Windstream!

    Share

Comments have been disabled for this post