Why are networks suddenly conking out all over the world? It looks like global networks are developing a signaling problem – more specifically a signaling overload problem.

Details are starting to emerge about just what caused the Orange and O2 outages. Computerworld UK and Information Age separately reported that the network element at fault in both cases was the home location register, or HLR. It’s not exactly the most commonly known piece of gear, but in brief the HLR acts as an anchor point to which we remain tethered as we move about the network. It stores our subscriber identities and knows what services we can access, but most importantly, it tracks each device’s present location so the network knows where to direct inbound and outbound traffic.

The HLR plays its dispatch role by receiving a constant stream of signals from devices updating the database on their current locations and activities.  According to Computerworld, a data glitch in an Orange HLR node generated error messages, which then multiplied as they got knocked back and forth around the network. Just because the HLR was failing, that didn’t stop devices from sending out their updates. Like a million kids screaming “look at me!” from the backseat while you’re trying to deal with the coffee you just spilled in your lap, smartphones kept pinging the suffering HLR creating a huge bottleneck. The end result: the whole system fails, leaving millions of handsets without their lifelines to the network core.

If the Orange and O2 failures sound familiar, it’s because the exact same thing that happened to Verizon in December. Since Verizon’s network is an LTE system, not an HSPA one, its core architecture is a bit different, but the basic problem seems to be the same. A software bug generated error messages that backed up its core elements, causing them to be oversaturated by signals and ultimately forcing the whole core to crash.

A whole lot of bandwidth but nowhere to go

In all three situations, the radio networks weren’t the problem. The networks still had plenty of capacity, and all devices were capable of connecting to their towers to send and receive data. But with a broken core, the networks had no idea where and whom to send that data to. Imagine playing Where’s Waldo? with 10 million people in a single storybook frame.

In Verizon’s case you could chalk it up to the relative newness of both the network and the LTE standard, but in the case of Orange and O2, their UMTS networks have been up and running for nearly a decade. For their HLRs to now start developing random terminal bugs seems rather odd. The problem doesn’t appear to be inherent in the equipment itself but in the sheer volume of signaling traffic traversing mobile networks driven by the smartphone boom.

That constant network chatter from smartphones and their applications are overwhelming network cores. On normal days they can handle that traffic, but even a small glitch throws everything out of whack. Smartphone use is only increasing, so this problem is only going to get worse.

What’s to be done?

If you talk to the signaling system vendors such as Tekelec, Acme Packet, Traffix Systems, Intellinet and Openet, you’ll get a single resounding answer: Diameter! Diameter is signaling protocol used in LTE core networks, and those aforementioned vendors claim that more robust and flexible routers using that protocol will nip the signaling problem in the bud. Diameter’s load balancing techniques would allow the network to shift the signaling load away from elements experiencing problems — isolating failures rather than allowing them to infect everything around them.

Given O2 and Orange’s failures, those vendors are jumping at the chance to claim diameter routers are now necessary for 3G networks as well, and they’re probably right. The vast majority of smartphone traffic currently runs through 3G towers, and it’s going to remain that way for a while. But diameter is by no means a cure-all.

Verizon has experienced a record number of network failures, even though its uses the next generation signaling protocol. despite the fact it implemented Tekelec’s diameter platform last year. Tekelec certainly isn’t to blame for the outages – they The outages were caused by software bugs in other elements, yet its diameter routers weren’t able to contain the problem, either, when the network started going haywire. Update: While Tekelec in August revealed that Verizon was a customer for its Diameter signaling router, Tekelec officials told me that Verizon hadn’t actually deployed its equipment by the time of the December outages.

Whatever the eventual cure, the wireless industry had better find it quick. O2’s London outage was particularly embarrassing because of the upcoming Olympics. But other operators should be just as worried. A network that needs to be shut down and rebooted every few months isn’t much of a network at all. 

Tower Image courtesy of Flickr user Nikhil Verma; Compass photo courtesy of Shutterstock user Sashkin

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• Forecast: the future of near field communication
• How to manage the signaling storm in 2013

Called the ASR5500, the core gateway is designed to handle what Cisco called the “new normal” of the mobile data network: millions of devices, each with dozens of applications in constant communication with the network. According to Murali Nemani, Cisco director of service provider mobility, that influx presents three distinct challenges: maintaining millions of sessions, supporting millions of individual transactions and handling enormous throughput.

You can distinguish between the three by breaking down a Google Chat conversation, Nemani said. Chat is constantly creating sessions over the network, pinging Google’s servers to see if an IM session is being initiated. If a chat session is opened, the messages sent between the participants are the transactions, while the actual payload of those messages – the text, video or voice – is handled by raw network throughput.

All of the background sessions and transaction signaling that makes our mobile apps tick run over the what’s known as the control plane, while what we actually experience – the services and content – ride the bearer plane. The capacity of both planes needs to be scaled enormously to meet the smartphone’s unquenchable hunger, but the problem facing carriers is that the capacity demands of those planes varies dramatically from site to site and time of day.

“In the morning there is a massive spike in signaling as people check e-mail and do social networking,” Nemani said. All of that signaling traffic places enormous demands on the control plane, but requires very little in terms of throughput, he said. “In the evening it’s throughput that’s in most demand as consumers shift to watching video and viewing other content,” Nemani said.

When designing the ASR5500 Cisco attempted to create an “elastic” core, which would prevent operators from having to scale both their signaling and throughput requirements for the worst-case scenarios of any given day, Nemani said. Rather than create dedicated resources for each type of traffic, the gateway’s processers are generic, allowing capacity to be shifted between the control and bearer planes in real-time. The result, Nemani said, is a dynamic mobile core that at any given moment can be signaling juggernaut or a packet-routing powerhouse.

Two carriers, Verizon Wireless and India’s Bharti, have already installed the new gateway in their networks, but given Cisco’s track record in the mobile core, many more operators are sure to follow.

While we often hear of the big radio network deals going to Ericsson and Nokia Siemens Networks, Cisco has been the unsung vendor in the guts of the network ever since its scooped up gateway specialist Starent in 2009. The core is a much smaller contract than the radio win in terms of revenues and it’s often less publicized. Synergy Research estimates that the mobile core market totaled $2.4 billion globally in 2011. It’s growing rapidly, but to put that in perspective a single large LTE radio network build can easily cost more than $2 billon.

Cisco, however, is by far the market leader when it comes to the mobile core. According to the vendor, it has won 270 3G total gateway contracts and is the core supplier for more than 30 LTE networks.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• New solutions for the evolving mobile network
• CES 2012: a recap and analysis