Anyone who has ever seen footage of a hurricane’s aftermath is probably now familiar with the COW, or Cell on Wheels. These temporary cell sites stand in for towers and base stations knocked out by storms (and boost mobile capacity at the Super Bowl), but what happens if a severe storm, earthquake or terrorist attack takes down a far bigger chunk of the communications network?
This week AT&T’s(s t) National Disaster Recovery team gave me a sneak peek of an exercise in Chicago to prepare for the most severe outage scenario in its network short of its Global Network Operations Center in New Jersey going offline. The drill was designed to explore how AT&T would deal with a disaster that took out an entire metro central office.
A central office may sound like an admin building, but in telco-speak it’s the term used for those huge windowless buildings packed to the gills with core infrastructure – the terminus for a carrier’s metro fiber lines and way station for all phone conversations. Knocking out a major CO like AT&T’s 27-story concrete monolith on Chicago’s Canal Street could leave much of the Windy City without a dial-tone or internet connection.
So what does AT&T do in case of such a disaster? It brings in a fleet of trucks hauling what amounts to a complete core network in tow. That means fiber trailers to reconnect a city’s downed optical lines, IP recovery trailers that house massive banks of routers, and power trucks and racks upon racks of generators to feed it all.
And because an event significant to take out a central office is probably going to do a lot of collateral damage to the network, that means mobile base stations: plenty of COWs, COLTS (cells on light trucks) and satellite uplinks to get emergency responders and the general populace back on the grid immediately. The teams that put all this together are regular AT&T employees, but they’ve all been trained for these disaster recovery scenarios, said AT&T Senior Network Specialist Kelly Morrison, who ran the Chicago exercise.
A major core outage doesn’t happen that often, but it has happened. After 9/11, AT&T’s downtown Manhattan network office suffered a complete failure. The National Disaster Recovery team had to recreate it across the Hudson River in Jersey City, attaching to the same fiber ring that served lower Manhattan.
Nobody is hoping for another disaster of such scale, Morrison said, but AT&T is prepared for outages of even bigger magnitude. The disaster recovery team has $600 million worth of emergency network equipment at its disposal – enough to build a nationwide communications grid in a small country – all of it distributed across the lower 48 states where it can be deployed quickly by truck or plane. Fully equipped, Morrison said, the team can assemble a temporary core network capable of handling 15 terabits per second of capacity.
