There is a fierce battle going on to control the future of the internet, and consumers are the innocent bystanders.
Reports have been filtering in during the last few days that consumers on Verizon (s vz) and Time Warner Cable’s (s twc) network are experiencing degraded service when they try to watch Netflix or YouTube videos. It may seem trite to whine about someone’s Arrested Development episode buffering, but the real issue is how big ISPs are trying to remake the agreements that underpin how the internet works.
As they do so, they are taking agreements that used to be negotiated by engineers based on web traffic and changing them into disputes negotiated behind closed doors in boardrooms, executives fighting over who has access to the end consumer. This could fundamentally change the way the internet works — making it more expensive to do business and erecting unnecessary barriers to innovation.
This is the new battle for the internet. The telcos and cable providers, intent on protecting their margins and their pay TV businesses, have taken network neutrality from the public world of consumer pricing and throttling to the data centers. Instead of banning Skype, or charging more for it on their networks, they want to change they way they charge content providers, demanding that they pay more for ports on the network when traffic starts filling them up.
The weapon in this battle is a concept known as peering. Peering is essentially an arrangement between two bandwidth providers — the companies that control the physical backbone of the internet — in which they send and receive traffic from each other for free. The logic is that the traffic sent from one network to another is reciprocated without adding extra costs and hurdles. This makes the web more efficient and redundant because companies don’t need to build out a network to connect every single service to every person who wants to consume that service.
There are about 50 major internet bandwidth providers around the world that connect to each other and work under that arrangement. There are about 5,000 other networks who then work with these major bandwidth providers to bring the internet to our homes, offices, iPads and iPhones. Of course, I’m being simplistic when I describe the notion of peering, but you get the gist. And this idea of peering — on which much of the modern internet is built — is coming under attack, thanks to ISPs like Verizon and Time Warner Cable that provide us (the consumers) with connectivity.
Two examples of this shift
Here’s how these battles tend to play out in the real world.
Picture a conference room in Arizona. In such a room last year, executives from Cable One, a cable broadband provider with 720,000 subscribers, asked the chief technology officer at one of world’s largest bandwidth providers to pay Cable One a fee each time it needed to add more ports to deliver content to Cable One’s end subscribers due to an increase in demand for certain types of content. The CTO of that bandwidth provider, who declined to be named, refused. He walked away rather than add a new, and to his mind, unsustainable, cost to the company’s bottom line. (A representative from Cable One did not immediately respond to a request for comment on this incident.)
Now consider another scenario, which is more in tune with how the internet currently works. Picture a data center in California, where Sonic.net, a Santa Rosa, Calif.-based independent broadband and phone company, also keeps its servers. Sonic.net, which has about 50,000 subscribers, sees a bit under 10 gigabits per second of Netflix traffic a day.
In order to deal with that traffic, Dane Jasper, the CEO of Sonic.net, has put a Netflix-designed box to cache Netflix content closer to the edge of the network where the ISP network meets the big bandwidth providers’ networks. This helps Sonic.net cut the cost that Netflix traffic can impose on its operations. Essentially, Netflix pays for its traffic to travel across most of the network before dropping it at Sonic.net’s door.
Telcos want to make the internet like the phone system
Open Connect, Netflix’s content delivery network, is a new wrinkle in the old idea of networking peering. But it tries to keep the same relationship, namely that both parties (those sending and those receiving content) share in the costs of delivering traffic to the end consumer. This model has worked for years, but some ISPs want to change it.
They see their pipes becoming commodified — dumb, as it were. So they want to build a new internet model: one based on terminating traffic that looks a lot like the old-school telephone networks, where ISPs serving the last mile can charge content companies and bandwidth providers for upgrades they make to the network in order to carry the increase in traffic. The point of connection and negotiation has become the ports where traffic hops from the bandwidth providers onto the ISPs’ network.
An OECD report on peering and internet exchange points from last October explained that because many areas of the world have efficient and cost-free peering relationships, the prices for data are around 100,000 times lower than the price of a voice minute.
As consumers, we are experiencing this underground tussle via poor video streaming and what are effectively throttled connections. For example, earlier this week Om and I wrote about Verizon’s refusal to work with Cogent because it was carrying traffic for a large video provider. Verizon was letting its connections to Cogent (multiple 10 gigabit per second ports) run hot — a euphemism for getting crammed with traffic. And that’s because Verizon has refused to open more ports — the equivalent of opening lanes of traffic on a highway — to Cogent.
Normally when this happens the two parties come to some sort of agreement about adding another port, and then the problem is solved. But in this case, Verizon doesn’t want to play. We don’t have details about what’s happening in the Verizon/Cogent spat, but we can share how these fights have played out in the past, thanks to a public peering squabble that happened in 2010.
We’ve been here before
On the final weekend of November 2010, engineers at Comcast were stunned to find themselves overloaded on ports that connected to Level 3 Communications. These ports — where Level 3 terminated its traffic destined for Comcast’s last mile subscribers — were getting slammed with 10 to 100 times the bits they normally saw. And since overloaded ports result in a crappy experience for the end customer, Comcast needed to figure out what was happening.
Engineers trying to determine where the huge influx of traffic was coming from finally settled on the culprit — a deal that Level 3 had signed earlier that month to carry traffic for streaming video service Netflix. At the time, Netflix traffic made up 33.7 percent of North American web traffic, and in that one weekend most of that traffic suddenly came into Comcast’s network from Level 3 as opposed to Akamai, Netflix’s previous content delivery partner.
The internet is flexible and redundant, but it’s still grounded in actual physical infrastructure. So when all those bits suddenly came into Level 3’s ports instead of the ones Comcast had designated for Netflix/Akamai traffic, packets started dropping and Netflix customers on Comcast network experienced poor service. It’s not all that different from when the police shut off access to a major highway and send people onto local streets: Traffic backs up and everyone’s unhappy.
Luckily, building new roads on the Comcast network isn’t as labor-intensive as building out roads in real life. Even so, when it came to adding new ports for Level 3, Comcast took the opportunity to try to renegotiate peering contracts with Level 3. “These fights over peering can be like third-graders squabbling on the playground,” said Patrick Gilmore, chief architect, network infrastructure for Akamai, in an interview last summer.
While the fuss over the Comcast and Level 3 spat eventually died down after the two parties agreed to some form of give-and-take on getting new ports on the Comcast network, it highlighted a basic fact of life on today’s internet. In many cases, as the big get bigger, the internet’s core weakness isn’t technical; it’s the fact that the main players are now so large that they are in the midst of negotiating a new balance of power.
It’s an IP version of the treaties and shifting alliances in Europe in the late 1800s that then fell apart during the first World War. And while lives aren’t exactly at stake in these peering battles, the outcome of these fights might change the way the internet works, making it more expensive to build a business on the internet and allowing ISPs to become a new gatekeeper.