When the network gets fried: A photographic look inside AT&T’s disaster recovery operations

Anyone who has ever seen footage of a hurricane’s aftermath is probably now familiar with the COW, or Cell on Wheels. These temporary cell sites stand in for towers and base stations knocked out by storms (and boost mobile capacity at the Super Bowl), but what happens if a severe storm, earthquake or terrorist attack takes down a far bigger chunk of the communications network?

This week AT&T’s National Disaster Recovery team gave me a sneak peek of an exercise in Chicago to prepare for the most severe outage scenario in its network short of its Global Network Operations Center in New Jersey going offline. The drill was designed to explore how AT&T would deal with a disaster that took out an entire metro central office.

Photo: AT&T

A central office may sound like an admin building, but in telco-speak it’s the term used for those huge windowless buildings packed to the gills with core infrastructure – the terminus for a carrier’s metro fiber lines and way station for all phone conversations. Knocking out a major CO like AT&T’s 27-story concrete monolith on Chicago’s Canal Street could leave much of the Windy City without a dial-tone or internet connection.

AT&T’s disaster recovery trailers in Chicago’s Soldier Field parking lot (Photo: Kevin Fitchard)

So what does AT&T do in case of such a disaster? It brings in a fleet of trucks hauling what amounts to a complete core network in tow. That means fiber trailers to reconnect a city’s downed optical lines, IP recovery trailers that house massive banks of routers, and power trucks and racks upon racks of generators to feed it all.

An IP recovery trailer packed with enough routing gear to handle 100 Gbps of IP traffic. (Photo: Kevin Fitchard)

A bank of batteries fed by a mobile power plant: All power has to be converted from AC to DC to protect against power surges. (Photo: Kevin Fitchard)

DS3 or T3 lines. Though much of AT&T’s traffic now travels over fiber, the disaster recovery team still has to restore older copper data connections. (Photo: Kevin Fitchard)

And because an event significant to take out a central office is probably going to do a lot of collateral damage to the network, that means mobile base stations: plenty of COWs, COLTS (cells on light trucks) and satellite uplinks to get emergency responders and the general populace back on the grid immediately. The teams that put all this together are regular AT&T employees, but they’ve all been trained for these disaster recovery scenarios, said AT&T Senior Network Specialist Kelly Morrison, who ran the Chicago exercise.

An AT&T National Disaster Recovery crew connecting power cables (Photo: AT&T)

AT&T’s Kelly Morrison with a Hazmat suit. A portion of the disaster team is trained to deal with hazardous materials so they can access contaminated facilities. (Photo: Kevin Fitchard)

An emergency communications van with satellite uplink, Wi-Fi and mobile small cell. Usually the first vehicle on-site during a disaster (Photo: Kevin Fitchard)

A major core outage doesn’t happen that often, but it has happened. After 9/11, AT&T’s downtown Manhattan network office suffered a complete failure. The National Disaster Recovery team had to recreate it across the Hudson River in Jersey City, attaching to the same fiber ring that served lower Manhattan.

The retracted tower mast of a COLT (Photo: Kevin Fitchard)

AT&T’s disaster recovery team will tap into local power supplies where available but can run its network off generator power if necessary (Photo: Kevin Fitchard)

Nobody is hoping for another disaster of such scale, Morrison said, but AT&T is prepared for outages of even bigger magnitude. The disaster recovery team has $600 million worth of emergency network equipment at its disposal – enough to build a nationwide communications grid in a small country – all of it distributed across the lower 48 states where it can be deployed quickly by truck or plane. Fully equipped, Morrison said, the team can assemble a temporary core network capable of handling 15 terabits per second of capacity.