Comments on: How good prep and a bucket brigade kept Peer 1 online during Hurricane Sandy http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/ Sun, 26 May 2013 06:11:05 +0000 hourly 1 http://wordpress.com/ By: Anonymous http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1140101 Thu, 01 Nov 2012 20:35:59 +0000 http://gigaom.com/?p=579136#comment-1140101 From http://forums.peer1.com/viewtopic.php?f=37&t=7532#wrap

06:30 PT – We are going to implement a controlled shutdown of NY Data Center at 10:45 ET. Customer communications is being prepped.

At this time an email went out to my company and I’m assuming most others at 10:25AM EST telling us we had 30 minutes to shut everything down:

“PEER 1 Hosting¹s local facilities team has been on-site at our NYC data center, maintaining our systems throughout the night while running on generator power. Due to the severe impact of this storm on the infrastructure in Manhattan, we are requesting all our customers hosted in our NYC data center to plan on gracefully shutting down your equipment between now and 11:00am EST today.”

I’m assuming hundreds of machines were powered down. Our company issued a controlled shutdown of all our servers there to avoid the impending abrupt power failure — including many that are mission critical to my business. The other option was potential data loss and/or hardware damage associated with an unplanned power loss / brownout / blackout.

I can only assume that this dramatic decrease in load contributed to subsequent forum posts about how the fuel was lasting “much longer than expected.”

Anyhow, we were down for quite some time, and still have many important machines we haven’t powered up yet. We turned on some machines after reading the hourly updates that “Generators are running and providing power.” Still, some mission critical data is dark — including a fairly large cluster — until we have confidence that the new fuel pump and AC issues have been worked out to the point that an unexpected blackout isn’t a likelihood.

]]> By: Michael http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1139847 Thu, 01 Nov 2012 18:40:03 +0000 http://gigaom.com/?p=579136#comment-1139847 Whatever they are doing beats Windstream!

]]> By: Heather Singer MacKay http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1139804 Thu, 01 Nov 2012 18:29:36 +0000 http://gigaom.com/?p=579136#comment-1139804 What impresses me about this story is that this is an amazing example of a true customer service culture, and so much so that customers are participating in the business continuity! Even if there have been some outages, as alleged in other comments, at this level of disaster, this is inspiring. Good for Peer1 staff!

]]> By: guest http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1136903 Thu, 01 Nov 2012 01:51:26 +0000 http://gigaom.com/?p=579136#comment-1136903 Are you sure it wasn’t actually an outage upstream from PEER 1? There are many companies that connect into PEER 1 and vice versa that have had and continue to have outages, which Peer can’t help. I’m not really sure its credible to say they had power-related downtime or downtime that affected their customers that was their fault. This would be a little more informative if you shared some details about your experience.

]]> By: Lee Colleton ★ http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1136188 Wed, 31 Oct 2012 20:31:59 +0000 http://gigaom.com/?p=579136#comment-1136188 “The Cloud” is tinged with diesel exhaust.

]]> By: Anonymous http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1136132 Wed, 31 Oct 2012 20:11:31 +0000 http://gigaom.com/?p=579136#comment-1136132 Inspiring, but the reality is that they haven’t maintained uptime for many of their customers during this window despite these above-and-beyond efforts. Bad situation, but the fact remains that this a true “outage” at Peer1 for many companies (including mine.)

]]> By: Barb Darrow http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1136121 Wed, 31 Oct 2012 20:08:17 +0000 http://gigaom.com/?p=579136#comment-1136121 right i know that Peer1 and other tenants — inap etc were looking at pumping alternatives…bucket brigade is certainly not meant to be a long-term fix.

]]> By: Lorenz Redlefsen http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1135857 Wed, 31 Oct 2012 18:48:39 +0000 http://gigaom.com/?p=579136#comment-1135857 If they have less powerful pumps, they could put one at street level, and another halfway up (say, the 8th or 9th floor), so that less powerful pumps can do the job. What matters is the height of the fuel column the pump is supporting.

Squarespace’s 12:40pm update says: “The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.”

]]> By: Henry Perkins http://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/#comment-1135113 Wed, 31 Oct 2012 15:48:54 +0000 http://gigaom.com/?p=579136#comment-1135113 Reblogged this on The Perkins Journal and commented:

This is so freaking cool.

]]>