Blog Post

3 lessons from the Blackout Bowl

It’s safe to say that the electrical systems supervisor is not the person that Superdome officials wanted the world’s media to be talking about the morning after the Super Bowl. For 34 dimly lit minutes, starting early in the third quarter, that person’s competence was one of the many things that the more than one billion people watching the game were discussing.

There’s a lot we don’t know about exactly what happened when the lights went out in the Super Bowl. But here’s what we do know:

Not all the lights went out: One-third of the lights stayed on throughout that excruciating half hour.  That probably means that the uninterruptable power supply system worked as planned.  The only problem was that the UPS system was sized to one-third the necessary power needs of the stadium.

The lights weren’t the only things going out:  The CBS announcers lost power, as apparently did the top-level cameras and the coaches’ communications systems.  This points to a failure in wiring the building’s critical circuits.  By far the most important thing to keep going in the case of an emergency (after emergency lighting and the PA system, both of which worked) is the power to the television operations.  Television is what pays everyone’s bills, so that should have priority over other systems.  It did not.  Likewise, the fact that one team’s communications systems continued to work (the 49ers’) and the other’s didn’t (the Ravens), showed that someone didn’t think very clearly when designing the critical circuit design.

LEDs Still Shone: If you looked carefully at the scenes of the blackened sections of the stadium seating, you could see that the emergency stair lights were all still lit.  Likewise, the exterior colored lighting that bathes the outside walls of the stadium in light was still working.  That’s because it’s made up of LEDs, which consume a fraction of a percentage of the power required by the sodium high intensity discharge (HID) lamps used for the rest of the stadium lighting.  Additionally, the sodium HIDs, once they went out, took another 20 minutes to regain their full luminosity.  LED’s, on the other hand, require no warm-up time and sip so little electricity that managing the current for them is a much less complex task.

Engineers & Repairmen

Based on this knowledge, here are three important lessons learned from the power management debacle that was Super Bowl XLVII:

  • Right-sizing a UPS backup microgrid is about more than just installing a bunch of generators.  The art of designing a backup microgrid is about balancing the maximum number of diesel gen-sets with the minimal amount of load.  Physical space for backup gen-sets is almost always limited (especially in a flood plain like New Orleans, where generators have to be placed – at a minimum – on the second floor).  Thus keeping the blackout from happening was more of a failure of critical circuit design than of generator management.
  • Energy efficiency counts more than backup power in times of emergency.  The failure of the sodium HID lights and the long warm-up time they require would have been solved by energy efficient LED lights, which also would have reduced the load on the UPS system.
  • Electrical design engineers are always more valuable than electric repairmen.  Designing the critical circuits to be prioritized during a power failure is a job worth doing right, as we saw on Sunday evening.  The designers of the Superdome’s UPS circuitry got some things very right: the success of the emergency lighting system kept the crowd from panicking.  But the problems with the broadcasting and team communications systems showed that not everything was so well-planned.

This article originally appeared on the blog of Pike Research. Pike Research, a part of Navigant Consulting’s global Energy Practice, is a market research and consulting team that provides in-depth analysis of global clean technology markets. Pike Research is also a partner of GigaOM Pro, GigaOM’s premium research service.

Image courtesy of delgaudm, Flickr creative commons.

11 Responses to “3 lessons from the Blackout Bowl”

  1. Sam Jaffe

    In addition to the above comment, I would also like to update what we do know about the electrical failure in New Orleans. In the second paragraph of the article, I wrote:

    “There’s a lot we don’t know about what happened when the lights went out in the Superdome…”

    Well now we do know quite a bit more, thanks to this excellent piece by the Times Picayune:

    It turns out that in December, the stadium staff bifurcated the high voltage line going in to the stadium into two medium voltage lines. Thus they had two separate electrical systems, one for each half of the building. Only one of those circuits failed. Thus the half of the lights that were on were still being fed by the grid, not by the UPS system. That’s also why the Ravens coaches (who were in a box on the dark side of the stadium) had no power and the 49ers coaches did. The UPS system did go on, but it fed only the emergency lighting system, some emergency signage and the P.A. system.

    Nevertheless, I would still argue that this event will cause stadium operators to seriously consider LED stadium lighting and a full microgrid. Yes, such systems cost an enormous amount of money. But stadium operators don’t make money by buying cheap infrastructural equipment. They make money by insuring that the show will go on. Especially when it comes to high-profile stadiums that bid against eachother for premium events, the existence of a microgrid that won’t fail is a major selling point. The cost of LED lights (which allow the microgrid to be significantly downsized to more affordable levels) are trivial if it means that a stadium wins the right to host the Super Bowl.

    • Earl Lee Popp

      Sam, you write “stadium operators don’t make money by buying cheap infrastructural equipment”

      But the truth is this: for most (though not all) of the events put on at major stadiums, there is little real impact to a short-duration power outage.

      It’s only one or two Superb Owl (remember, we can’t legally use the name of that event) type spectacles for which the PR damage is significant enough to warrant considering a full power systems retrofit.

      Many (if not most) large stadiums today are publicly owned and financed (as the Superdome is) and so you’re asking the taxpayer to foot the bill for many billions of dollars of new infrastructure across at least a dozen big-event qualified stadiums nationally just to satisfy the needs of a single event holder?

      That doesn’t make sense to me. Especially when that event holder is a multi-billion dollar non-profit company (yes, the NFL as a league is a non-profit, though the individual team franchises are for-profit companies).

      This to me is a situation where the status quo is probably good enough. If it isn’t, then it’s likely incumbent on the NFL to pony up the dollars to boost the infrastructure of its chosen host stadium. After all, they select the stadiums 5 years in advance, so the league can definitely work on the power issue long before game day.

    • This article is an excellent example of:
      1) the demise of quality real journalism as the likes of the post, times etc. lose readers, and those readers turn to anyone with a computer blog for information. The blogoshpere is populated by anyone, who can say anything about anything with no requirement for credibility or knowledge. Blog “information” is nothing better than what you hear “around the water cooler”. Often the content is created solely to sell ones services or companies products. There is no independence or objectivity guaranteed by the millions of voices.
      2) Sam Jaffe, as grammatically correct as the article may have been, apparently knows very little about power distribution, either outside or inside facilities, nor the architecture of power for broadcast events. Every commenter thankfully could sense that. Sadly many people read and believed at face value what was presented.

      The fact is, the article Sam latter posted a link to has the most accurate info of all, and it is entirely articulated in this single sentence: “”It was a piece of equipment that did its job, {Doug Thornton, senior vice president of SMG, which manages the Superdome} We don’t know anything beyond that. It’s premature at this point to say what it was or what caused it.”

      And that was they key take way or to use Sam’s headline: “Lesson”. The only “Lesson” to be learned from this article is not to write speculative articles making up things before the facts have been collected. Do your research, or make it clear though your grammar that you are entirely speculating and guessing.

      Lest you undermine your own credibility.

      Keep that in the back of your mind anytime you read an “article” on blog or anywhere else on the net.


      said. “We don’t know anything beyond that. It’s premature at this point to say what it was or what caused it.”


  2. Sam Jaffe

    I’m having a hard time understanding where your comments intersect with the column that I wrote. Are you sure you’re not commenting on someone else’s piece of writing? Nowhere in the piece did I suggest that the LED emergency lights were attached to the UPS system because of the physics of the lightbulb themselves. I was pointing out that because they are LED’s, the designers were able to put a lot more in–because they sip so little energy. In fact, I would guess that they did so on a low-voltage line, which would have reduced costs for the entire emergency lighting system significantly. That led to my bigger point about LED stadium lighting: its use would allow the entire lighting system to go on the UPS system, thereby eliminating the problem they had on Sunday night.

    As to my point about critical circuit design, let me try to re-articulate it because I don’t think I was able to make myself understood: the stadium electrical system designers defined an emergency event as a mass panic that would lead to people trampling each other as they fled for the exits. Thus they sized the UPS system to avoid that (giving power to the P.A. and emergency lighting system as planned). The mistake they made was in the definition of “emergency event”. In fact, the sodium lights going out was an emergency that will probably end up costing tens of millions of dollars in lawsuit compensation (I would bet that CBS will be suing them because they will lose a lot of money because of slotting guarantees for commercials; I would also bet that the case will probably be settled out of court quickly because, frankly, the Superdome officials don’t appear to have a very good case, based on what we’re learning now about engineers’ prior warnings). To the operators of an entertainment business, the definition of an emergency should include not screwing up the most significant entertainment event of the year. Thus putting the entire system on a microgrid that is affordable because it only has to handle the load of LED lights starts to be worthy of being brought up.

    I’m not an IT professional. I don’t understand what you mean by saying that “smart grid” has something to do with what happened in the Superdome. Smart grid, as its name implies, is about the electricity distribution grid. The point of failure in the superdome happened behind the meter, making discussions about what the smart grid could have done for the Superdome kind of moot.

    Charles McReary:
    Here’s a video of the launch of Ephesus’ first project in upstate New York. You could have also clicked on the link in the article itself.

    • Gary Whittaker

      I don’t know what they are talking about. I saw one reporter who was outside at the time filming and the outside lighting did go out while she was talking.

  3. Talk about a lack of knowledge about electrical and lighting. Republishing this trash was a mistake on GigaOm’s part.

    Not all the lights went out. Virtually every physical plant on the planet only wires critical systems to UPS/Generator because of the enormous cost. The facility was sufficently lit to manage an exit of the facility, which is exactly what it was designed to do.

    The lights weren’t the only thing that went out. See previous about critical systems. I was actually impressed at how many cameras continued to operate considering the fact that CBS used 3 times as many cameras as is used in a regular season game. I’ll give you a point that the booth should have been on redundant power, but that is more on the network guys than the plant guys.

    LEDs still shone. I almost fell out of my chair on this one. Efficiency will not keep a light burning without power. The stair lights were burning because they were considered critical to emergency evacuation and on redundant power, per code, not because they were LED. Every little EXIT sign you see in the US, LED or otherwise, either has an internal battery or is wired to redundant power. I don’t know why the exterior lighting continued to work. High Pressure lamps use a ballast that, by design, must cool down before they can be restarted and they do take several minutes to move to full intensity. You argue that using LEDs instead of HPS would have allowed more lighting to be on UPS, but that leads to the final topic.

    Using LEDs instead of High Pressure Lamps. I think you need to go do some research. Throwing lots of light over large distances requires a very bright light source and the ability to focus that source into a tight beam. A typical High Pressure Lamp generates 100K+ lumens and uses a reflector to focus the omnidirectional light to a single direction. By contrast, some of the best LEDs are less than 1K lumens and have to be grouped to achieve higher output, but you eventually reach a point where the ability to shed heat is less than the lumens generated and LEDs hate heat. I have seen some pro lights that reach 30K, but not much higher. Also LEDs are inherently unidirectional so focusing multiple sources is extremely difficult, so throwing light further than a 100 feet with LED is pretty challenging. The lights on the Superdome are much further away. Finally, HPS is still significantly cheaper than LED and more tolerant of temperature changes.

    Basically you tried to apply your knowledge about Data Centers to a physical plant with an entirely different set of requirements and in the process showed your ignorance. This facility will only be used one more time in Feb and only 12 times in March (based on their published calendar) and most of these events would be merely inconvenienced by the outage…just as the Super Bowl was. In fact, you could argue that it generated revenue for the network because they were able to show more ads.


    • Fully agree. So happy someone posted a sensable reply. I fear this is indicative of a larger issue that is a misconception of IT professionals to think they understand the “smart grid” because it involves putting systems on a network. Your post highlights most poignantly that understanding business requirements requires an understanding far beyond the application of IT best practices.

    • Jesse Pink

      You obviously know a bit more about this than me, but even with my limited knowledge of building design, I knew most of what he said here was complete and utter BS.

      What would be the purpose of putting all the cameras on an emergency circuit?