In February, Oracle announced it would buy VoIP signaling vendor Acme Packet for $1.7 billion. The terms of the Tekelec deal weren’t disclosed. Once it closes on both investments, Oracle is set to become a signaling powerhouse.

Tekelec specializes in the signaling protocols and load balancing technologies that prevent mobile networks from getting overloaded. For instance, the outages Verizon experienced on its LTE network in late 2011 were partially attributable to signaling overload. Meanwhile, Acme Packet makes session border controllers (SBCs), which manages VoIP and multimedia control traffic that pass between carrier and enterprise networks.

Oracle, however, will get more out of Tekelec than just signaling expertise. Tekelec is also a big player in the traffic-shaping world. Mobile operators use its policy servers to prioritize bits from certain type of applications – and certain subscribers’ – over others. The result is a bunch of things most of you don’t often find pleasant, such as throttling back your data speeds when you exceed your monthly cap or detecting when you use your phone as a mobile hotspot and charging you extra for it.

But eventually those same policy management features will be used for a much broader range of features and tailored data plans. Jasper Wireless is using Tekelec traffic shaping technology to make data flow more smoothly in the internet of things, for example. And operators are weighing new types of tiered data plans that allow customers to customize their network connections based on the types of apps they use.

“Oracle has in the past partnered to provide these capabilities, but by bringing them in-house it will have more opportunity to shape the roadmap and combine the capabilities in a more tightly-coupled solution,” Ovum Principal Analyst Dana Cooperson said in a research note. “Expect Oracle’s telecom-focused competitors (Alcatel-Lucent, Huawei, Ericsson, etc.) and it’s IT-focused competitors (HP, SAP, SAS Institute) to do more strategic soul-searching and, as their financial situation allows, to pursue acquisitions of their own.”

Many of them already have. Cisco Systems bought policy management firm BroadHop in December, while Citrix System acquired ByteMobile.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• New solutions for the evolving mobile network
• Research In Motion: future scenarios for its fate
• Mobile 2012 and beyond

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Mobile 2012 and beyond
• CES 2012: a recap and analysis
• New solutions for the evolving mobile network

Why are networks suddenly conking out all over the world? It looks like global networks are developing a signaling problem – more specifically a signaling overload problem.

Details are starting to emerge about just what caused the Orange and O2 outages. Computerworld UK and Information Age separately reported that the network element at fault in both cases was the home location register, or HLR. It’s not exactly the most commonly known piece of gear, but in brief the HLR acts as an anchor point to which we remain tethered as we move about the network. It stores our subscriber identities and knows what services we can access, but most importantly, it tracks each device’s present location so the network knows where to direct inbound and outbound traffic.

The HLR plays its dispatch role by receiving a constant stream of signals from devices updating the database on their current locations and activities.  According to Computerworld, a data glitch in an Orange HLR node generated error messages, which then multiplied as they got knocked back and forth around the network. Just because the HLR was failing, that didn’t stop devices from sending out their updates. Like a million kids screaming “look at me!” from the backseat while you’re trying to deal with the coffee you just spilled in your lap, smartphones kept pinging the suffering HLR creating a huge bottleneck. The end result: the whole system fails, leaving millions of handsets without their lifelines to the network core.

If the Orange and O2 failures sound familiar, it’s because the exact same thing that happened to Verizon in December. Since Verizon’s network is an LTE system, not an HSPA one, its core architecture is a bit different, but the basic problem seems to be the same. A software bug generated error messages that backed up its core elements, causing them to be oversaturated by signals and ultimately forcing the whole core to crash.

A whole lot of bandwidth but nowhere to go

In all three situations, the radio networks weren’t the problem. The networks still had plenty of capacity, and all devices were capable of connecting to their towers to send and receive data. But with a broken core, the networks had no idea where and whom to send that data to. Imagine playing Where’s Waldo? with 10 million people in a single storybook frame.

In Verizon’s case you could chalk it up to the relative newness of both the network and the LTE standard, but in the case of Orange and O2, their UMTS networks have been up and running for nearly a decade. For their HLRs to now start developing random terminal bugs seems rather odd. The problem doesn’t appear to be inherent in the equipment itself but in the sheer volume of signaling traffic traversing mobile networks driven by the smartphone boom.

That constant network chatter from smartphones and their applications are overwhelming network cores. On normal days they can handle that traffic, but even a small glitch throws everything out of whack. Smartphone use is only increasing, so this problem is only going to get worse.

What’s to be done?

If you talk to the signaling system vendors such as Tekelec, Acme Packet, Traffix Systems, Intellinet and Openet, you’ll get a single resounding answer: Diameter! Diameter is signaling protocol used in LTE core networks, and those aforementioned vendors claim that more robust and flexible routers using that protocol will nip the signaling problem in the bud. Diameter’s load balancing techniques would allow the network to shift the signaling load away from elements experiencing problems — isolating failures rather than allowing them to infect everything around them.

Given O2 and Orange’s failures, those vendors are jumping at the chance to claim diameter routers are now necessary for 3G networks as well, and they’re probably right. The vast majority of smartphone traffic currently runs through 3G towers, and it’s going to remain that way for a while. But diameter is by no means a cure-all.

Verizon has experienced a record number of network failures, even though its uses the next generation signaling protocol. despite the fact it implemented Tekelec’s diameter platform last year. Tekelec certainly isn’t to blame for the outages – they The outages were caused by software bugs in other elements, yet its diameter routers weren’t able to contain the problem, either, when the network started going haywire. Update: While Tekelec in August revealed that Verizon was a customer for its Diameter signaling router, Tekelec officials told me that Verizon hadn’t actually deployed its equipment by the time of the December outages.

Whatever the eventual cure, the wireless industry had better find it quick. O2’s London outage was particularly embarrassing because of the upcoming Olympics. But other operators should be just as worried. A network that needs to be shut down and rebooted every few months isn’t much of a network at all. 

Tower Image courtesy of Flickr user Nikhil Verma; Compass photo courtesy of Shutterstock user Sashkin

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• Forecast: the future of near field communication
• How to manage the signaling storm in 2013

Some have pointed the finger at reluctant middle managers, but on EconLog this week economist Bryan Caplan reports that a new paper offers another compelling explanation. The senior paper by Georgetown undergraduate Alexander Clark, which Caplan, describes as “fascinating,” says the problem with the uptake of telecommuting is signaling. Caplan writes:

Workers physically commute for signaling reasons.  Employers can monitor your productivity better when you actually come to the office.  Workers who telecommute put themselves on the slow track to success – if they can even get hired in the first place.  To bolster this thesis, Clark analyzes the American Time Use Survey using the employer learning-statistical discrimination (EL-SD) framework.  He finds that the labor market does indeed take longer to reward telecommuters for their hard-to-observe abilities.

Caplan also offers an excerpt of Clark’s paper, which argues not only that managers fear telecommuters are shirking, making face time a signal of hard work, but also notes that showing up at the office, in essence, reaffirms a worker’s status as belonging to what amounts to the office tribe. Don’t show up and your boss and colleagues could take the move as a rejection. Clark writes:

In a recent Ipsos/Reuters poll, which questioned 11,383 people in 24 countries, about half believed that they would be at a disadvantage in earning promotions because of the lack of face-to-face contact (2012). Previous research suggests part-time telecommuters do not communicate less frequently with managers (Duxbury and Neufeld 1999). Even so, more than simple communication matters. Showing up at an office may signal positive attributes to a boss. If a boss leaves work for the day and notices an employee staying late, it could serve as a visual reminder of work ethic. Working in a shared workplace also gives greater opportunity to demonstrate cooperativeness. The employee recruitment process often emphasizes the importance of labels like “team player,” and many companies strive to create collegial work environments and attractive office cultures. If a boss were to psychoanalyze an employee’s decision to telecommute, the resulting signal likely would not be that the employee wants to use time saved commuting to put in additional work. At worst, telecommuting would be seen as an atomistic rejection of the (sometimes carefully constructed) office environment.

The effect of these signals was clear when Clark combed through the numbers. “After four years of experience, the average male telecommuter will earn about 6.9 percent less than a non-telecommuter,” concludes Clark.

Do you find the idea that signaling is at the heart of telecommuting’s anemic uptake convincing?

Image courtesy of Flickr user chidorian. 

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Future of Work Platforms: An Overview
• Personal tools lead to practical business
• Millennials in the enterprise, part 1: strategies for supporting the new digital workforce

According to Verizon’s tweet, only the 4G network appears to be affected:

VZW is investigating customer issues in connecting to the 4GLTE data network. 3G data, voice and text services are operating reliably.

Update: At 10:21 PT, Verizon notified customers through its Twitter feed that LTE service had been fully restored. Verizon called the outage a “brief issue,” which may be true, but it certainly wasn’t a localized one. As you can see in the comments to this post, GigaOM readers reported losing 4G connectivity, and sometimes 3G, from all over the country.

Depending on the scope and duration of the problem, this outage could pass by with little notice, or it could be another black eye for Big Red, which suffered a chain of big LTE failures in December. Those problems were all caused by bugs in Verizon’s core service delivery architecture – in telco speak called the IP Multimedia Subsystem, or IMS – but Verizon VP of network engineering Mike Haberman said the carrier had since taken multiple steps to ensure that such problems wouldn’t occur again. From our December post:

While Verizon won’t promise that no more outages will occur, Haberman said it has taken measures to ensure that they’re minimized when they do happen in the future. He said he’s begun geographically segmenting the LTE network, so if a software bug does break out it can be isolated to a particular region or market instead of spreading nationwide. Verizon is also upgrading all of its software and cutting down on the signaling clutter running over its IMS grid.

“Our goal is to ensure that our 4G network meets the same high standard that our 3G network does,” Haberman said. “We’re not there yet, but we’ll get there.”

In the current outage, 3G service appears to be working for LTE smartphone customers as it did in previous outages. Thus, Verizon may  be experiencing problems with its IMS core once again.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• LTE-Advanced: what it is and isn’t
• Sprint’s tightrope walk: finding a balance for its network modernization plan
• Mobile Q1: All Eyes on Tablets, T-Mobile and AT&T

The company said Wednesday that it raised a first round of $7 million led by Bessemer Venture Partners to help it expand operations. Ben Volkow, the CEO, says that 60 percent of operators already have some Traffix gear on their networks to address and manage signaling traffic, but more operators are interested. As more operators began deploying LTE networks, which add to the complexity of signaling traffic and to the overall network, Volkow decided that his previous strategy of growing the business through bootstrapping it no longer made sense. “We needed to scale and grow the business,” he said in an interview.

Signaling traffic is the data the phone or device sends out to the network to tell it where it is, what is it is doing, how much it is allowed to do based on the subscriber’s plan and figure out when to hop to the next base station. Chetan Sharma, a wireless analyst, issued a report last year noting that network congestion is generally caused by two big things: (1) signaling traffic caused by smartphones and superphones and (2) peak data traffic caused by data cards and embedded laptops.

He wrote that signaling traffic is growing faster than raw data traffic because smartphones are not very efficient with applications. As proof, he showed that smartphone signaling traffic is more than eight times data card signaling traffic, even though smartphones were only a small segment of the overall base of devices on the network. And this report was issued before smartphones had achieved the popularity that they have today!

At the time, he said signaling consumes more than 50 percent of available network resources. On LTE networks the problem intensifies as people use more mobile bandwidth on more devices and because the applications people run tend to talk back to the network more often. But when handled correctly, signaling traffic can add intelligence to the network and let operators route their mobile traffic more efficiently, perhaps by offloading it onto a nearby Wi-Fi hot spot or a picocell. Traffix sees an opportunity here, and has raised the money to take it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How to manage the signaling storm in 2013
• LTE-Advanced: what it is and isn’t
• LTE changes everything; LTE changes nothing