5 Comments

Summary:

Following the power outage during Sunday’s Super Bowl game, an Amazon Web Services engineer gave his blog readers some tips for avoiding similar events, at data centers or giant stadiums.

powergrid17

When a power outage affects a major television event, such as the Super Bowl, it’s a big deal. More than 108 million people were watching. Plus, in the case of Sunday’s game between the San Francisco 49ers and the Baltimore Ravens, the outage gave the 49ers a chance to rally after a tough first half.

But maybe sports facilities — and really anyone that wants to deliver a service — be it online music or a few hours of entertainment during a national sporting event — should start thinking about their infrastructure and their resiliency. After all, our infrastructure isn’t getting any younger, and consumers certainly aren’t getting any more patient.

That was the point of a blog post Amazon Web Services Distinguished Engineer James Hamilton wrote a day after the game. Hamilton offered up a few smart solutions for power outages. At least one of them — using diesel generators to ensure redundancy — makes great financial sense for something like the Super Bowl.

Two bus-sized 2.5-3 MW generators should be more than enough to back up the power supply from the Superdome’s utility, Entergy, Hamilton wrote. Each generator costs less than $1 million, Hamilton wrote, and can be rented for less. Add in switchgear, uninterruptible power supplies and a one extra generator for good measure, and the cost comes in at less than $10 million. Hamilton continues:

“Looking at statistics from the 2012 event, a 30 second commercial costs just over $4m. For the price of just over 60 seconds of commercials the facility could (be) protected against fault. And, using rental generators, less than 30 seconds of commercials would provide the needed redundancy to avoid impact from any utility failure. Given how common utility failures are and the negative impact of power disruptions at a professional sporting event, this looks like good value to me.”

Hamilton also suggests in the post that facility operators frequently test equipment, use automated recovery, use more utility breakers to decrease the amount of a facility an outage could wipe out and simply steer clear of sites in fault zones altogether. In short, big facility operators might want to think more like the cloud guys.

You’re subscribed! If you like, you can update your settings

  1. Doesn’t a power outage only serve to provide more room for commercials? Each 30 seconds they can not play football equals another $4m in potential ad revenue…

  2. At the same time all the stadiums at the African Cup of Nations 2013 have 200% power redundancy.
    To me what seems odd is that power redundancy is not mandatory after 9/11, with stadiums being possible targets a blackout would facilitate an attack and increase panic.

  3. Do you have rental numbers? I guess 2 weeks is minimum to allow for installation and testing.

  4. Nice thought, but generators would have done little to solve the problem. The utility (Entergy) says that at all times their feeds to the building were live. If the failure was faulty equipment within the building, then back up generators would have been equally useless. The fact that power was on to half the building suggests a failure of internal controls more than a power outage from the utility.

    While redundant power would be nice, it is expensive to maintain and severely hampered by emissions rules on when it can run. As a practical matter, backup power is more for maintaining critical systems. In this case, there was some disruption of people moving equipment, again suggesting a failure in design internal to the building.

    On the other hand, if the risk to revenue for the commercial enterprises justifies their investment in back up generation (again, new EPA rules at play), that is a business decision, not one on public safety. The players said afterward that there remained enough light to play physically, but the electronic aids were disrupted.

  5. Maybe a ClampStar could’ve prevented the power outage. At a cost of only $97 this unit would definitely have prevented the Dec. 2001 Candlestick Park blackout. And when you look at all the other power outages taking place every day in this country you gotta ask yourself, “in this day and age, with access to all kinds of advanced technology why do we have ANY power outages?” Of course, you’ll need to exclude those caused by mother nature but to lose power on a clear day with virtually no wind or bad weather is usually attributed to lack of proper preventive maintenance.

    In fact, preventive maintenance is so bad at almost all electric utilities in this country that every year people die from falling power lines and in almost every case it could’ve been prevented if proper maintenance was performed.

    Check this out, did you know that a triple electrocution took place in Jan. 2011 in San Bernardino, CA? It was caused by a downed power line that started a fire and when the father attempted to put out the fire he was electrocuted, and his wife was electrocuted when she came out to help him and then their son was electrocuted when he came out to help his mother and father. Does anyone know what caused that line to come down? Shouldn’t the utility be required to notify the public of the cause? Could it have been prevented? Absolutely!

    Unfortunately, once the lawyers take over and the victim’s family receives the pay-off the public remains in the dark and the utility continues with business as usual. Of course, they’ll raise their rates to compensate for the financial loss but will they change their preventive maintenance procedures to prevent this from happening again? History shows us that the answer is NO!

    That’s just one example. There are many more. I think the most recent downed power line catastrophe that became public knowledge was the one that took place in Irwin, Pennsylvania. Carrie Goretzka, 39, was inside her home on June 2, 2009, with her two daughters, then 2 and 4, and her mother-in-law when the power went out. It was a clear afternoon. She looked outside and saw trees in her yard on fire. Mrs. Goretzka retrieved her cell phone from her car and stood in the side yard to call 911 when the line fell on her. The electrified wire caused the woman to catch on fire. Her mother-in-law tried to help but was burned. It took 20 minutes before crews arrived and shut off the power.

    It was determined later that the cause of the downed line was due to a failed in-line splice. Had the utility placed a $97 ClampStar over that splice, Carrie Goretzka would be alive today. And the only reason we know the cause of this tragedy is because the case went to trial. This is very rare. Almost all of these types of cases are settled out of court but the fact that the public is now aware of this utility’s incompetence they will be required to correct all the overhead in-line splices in their system. This is how it should be handled in all power failure cases. Identify the root cause and prevent it from happening again!

    The present system prevents the public from knowing what really causes these catastrophes and that must change! Utilities need to be held accountable. With millions of miles of overhead lines in this country there’s a huge risk to public safety and the utilities should not be allowed to sweep their mistakes under the rug.

Comments have been disabled for this post