Verizon Wireless(s vz)(s vod) experienced another outage on its LTE network on Wednesday, causing customers across the country to lose access to their ultra-fast mobile broadband connections and deal with the slower data speeds of Verizon’s 3G network. Verizon restored the network to normal late Wednesday, but this latest glitch shows Verizon’s one-year-old 4G rollout is still experiencing some growing pains. As the first to launch LTE on a large scale, Verizon has become a guinea pig for the entire industry, making it the first operator to encounter and deal with the bugs remaining in the new technology.
Unlike the major outage Verizon experienced in April, Wednesday’s LTE problems didn’t affect all customers and it affected them intermittently. April’s outage affected all high-speed data connections to 4G customers; new HTC Thunderbolt and MiFi owners couldn’t even get a 3G connection and had to resort to the dial-up-modem-like speeds of Verizon’s 2G network.
This time around, customers affected couldn’t link up with the LTE network, and while there were reports of some customers losing their 3G access as well, most devices were able to fall back on Verizon’s 3G EV-DO network. EV-DO doesn’t pack the punch of LTE, but is enough to support most of what you would want to do on a smartphone. Verizon issued this statement on Thursday morning:
“Verizon Wireless 4GLTE service returned to normal Wednesday evening after the company’s network operations team resolved a technical issue. Some 4G customers had reported intermittent or unavailable 4G service, or devices operating on 3G, for periods of time starting late Tuesday. Throughout this time, all customers were able to make voice calls, send and receive text messages, and 3G data devices operated normally.”
In April’s outage, a software glitch originating deep within Verizon’s network core began knocking down elements of Verizon’s new LTE service delivery architecture, taking down the servers that identified individual customers as they moved through the network. That kicked all of Verizon’s 4G customers off both Verizon’s data networks. CDMA-only customers weren’t affected, because Verizon manages them on its older CDMA service delivery systems. This time, the problem doesn’t appear to have originated in the core, but rather in the radio network itself, otherwise, 3G data would have been knocked out as well. Or Verizon may have learned its lesson from that last outage and figured out a way to shunt 4G customers over to the 3G core, where their devices would behave like ordinary CDMA phones.
When Verizon built its new LTE network, it didn’t just mount new fatter radios on its towers; it changed the fundamental design of every aspect of its network: moving from hardware to software driven base stations, evolving its network service delivery systems from old hierarchical voice-centric chains of gateways to new flat IP architectures, and replacing old copper backhaul links with fiber Ethernet to the tower. It was also the first operator to do such a complete overhaul on such a huge scale. For many of the vendors in its network, Verizon’s early 4G launch marked the first time their new LTE equipment was deployed commercially. I’m actually surprised more bugs haven’t reared their ugly heads in the year the LTE network has been online.
Still, Verizon’s customers probably won’t cut the carrier too much slack. Verizon, after all, bills itself as having the country’s “most reliable network.” In April, the repercussions from the nationwide outage weren’t too bad, probably because Verizon only had a few hundred thousand LTE customers at that point. Today, it has several million. What happens when an outage occurs and it has tens of millions of 4G customers?