12 Comments

Summary:

The latest fight between ISPs and over-the-top providers is taking place deep in the network, away from the eyes of regulators and consumers. Welcome to the world of peering fights.

internetmapglobal
photo: Peer1

There is a fierce battle going on to control the future of the internet, and consumers are the innocent bystanders.

Reports have been filtering in during the last few days that consumers on Verizon and Time Warner Cable’s network are experiencing degraded service when they try to watch Netflix or YouTube videos. It may seem trite to whine about someone’s Arrested Development episode buffering, but the real issue is how big ISPs are trying to remake the agreements that underpin how the internet works.

As they do so, they are taking agreements that used to be negotiated by engineers based on web traffic and changing them into disputes negotiated behind closed doors in boardrooms, executives fighting over who has access to the end consumer. This could fundamentally change the way the internet works — making it more expensive to do business and erecting unnecessary barriers to innovation.

AT&T NOC HQ

This is the new battle for the internet. The telcos and cable providers, intent on protecting their margins and their pay TV businesses, have taken network neutrality from the public world of consumer pricing and throttling to the data centers. Instead of banning Skype, or charging more for it on their networks, they want to change they way they charge content providers, demanding that they pay more for ports on the network when traffic starts filling them up.

The weapon in this battle is a concept known as peering. Peering is essentially an arrangement between two bandwidth providers — the companies that control the physical backbone of the internet — in which they send and receive traffic from each other for free. The logic is that the traffic sent from one network to another is reciprocated without adding extra costs and hurdles. This makes the web more efficient and redundant because companies don’t need to build out a network to connect every single service to every person who wants to consume that service.

There are about 50 major internet bandwidth providers around the world that connect to each other and work under that arrangement. There are about 5,000 other networks who then work with these major bandwidth providers to bring the internet to our homes, offices, iPads and iPhones. Of course, I’m being simplistic when I describe the notion of peering, but you get the gist. And this idea of peering — on which much of the modern internet is built — is coming under attack, thanks to ISPs like Verizon and Time Warner Cable that provide us (the consumers) with connectivity.

peeringideal

Two examples of this shift

Here’s how these battles tend to play out in the real world.

Picture a conference room in Arizona. In such a room last year, executives from Cable One, a cable broadband provider with 720,000 subscribers, asked the chief technology officer at one of world’s largest bandwidth providers to pay Cable One a fee each time it needed to add more ports to deliver content to Cable One’s end subscribers due to an increase in demand for certain types of content. The CTO of that bandwidth provider, who declined to be named, refused. He walked away rather than add a new, and to his mind, unsustainable, cost to the company’s bottom line. (A representative from Cable One did not immediately respond to a request for comment on this incident.)

Now consider another scenario, which is more in tune with how the internet currently works. Picture a data center in California, where Sonic.net, a Santa Rosa, Calif.-based independent broadband and phone company, also keeps its servers. Sonic.net, which has about 50,000 subscribers, sees a bit under 10 gigabits per second of Netflix traffic a day.

In order to deal with that traffic, Dane Jasper, the CEO of Sonic.net, has put a Netflix-designed box to cache Netflix content closer to the edge of the network where the ISP network meets the big bandwidth providers’ networks. This helps Sonic.net cut the cost that Netflix traffic can impose on its operations. Essentially, Netflix pays for its traffic to travel across most of the network before dropping it at Sonic.net’s door.

Telcos want to make the internet like the phone system

Open Connect, Netflix’s content delivery network, is a new wrinkle in the old idea of networking peering. But it tries to keep the same relationship, namely that both parties (those sending and those receiving content) share in the costs of delivering traffic to the end consumer. This model has worked for years, but some ISPs want to change it.

more network cables

They see their pipes becoming commodified — dumb, as it were. So they want to build a new internet model: one based on terminating traffic that looks a lot like the old-school telephone networks, where ISPs serving the last mile can charge content companies and bandwidth providers for upgrades they make to the network in order to carry the increase in traffic. The point of connection and negotiation has become the ports where traffic hops from the bandwidth providers onto the ISPs’ network.

An OECD report on peering and internet exchange points from last October explained that because many areas of the world have efficient and cost-free peering relationships, the prices for data are around 100,000 times lower than the price of a voice minute.

As consumers, we are experiencing this underground tussle via poor video streaming and what are effectively throttled connections. For example, earlier this week Om and I wrote about Verizon’s refusal to work with Cogent because it was carrying traffic for a large video provider. Verizon was letting its connections to Cogent (multiple 10 gigabit per second ports) run hot — a euphemism for getting crammed with traffic. And that’s because Verizon has refused to open more ports — the equivalent of opening lanes of traffic on a highway — to Cogent.

Normally when this happens the two parties come to some sort of agreement about adding another port, and then the problem is solved. But in this case, Verizon doesn’t want to play. We don’t have details about what’s happening in the Verizon/Cogent spat, but we can share how these fights have played out in the past, thanks to a public peering squabble that happened in 2010.

We’ve been here before

On the final weekend of November 2010, engineers at Comcast were stunned to find themselves overloaded on ports that connected to Level 3 Communications. These ports — where Level 3 terminated its traffic destined for Comcast’s last mile subscribers — were getting slammed with 10 to 100 times the bits they normally saw. And since overloaded ports result in a crappy experience for the end customer, Comcast needed to figure out what was happening.

Engineers trying to determine where the huge influx of traffic was coming from finally settled on the culprit — a deal that Level 3 had signed earlier that month to carry traffic for streaming video service Netflix. At the time, Netflix traffic made up 33.7 percent of North American web traffic, and in that one weekend most of that traffic suddenly came into Comcast’s network from Level 3 as opposed to Akamai, Netflix’s previous content delivery partner.

The internet is flexible and redundant, but it’s still grounded in actual physical infrastructure. So when all those bits suddenly came into Level 3′s ports instead of the ones Comcast had designated for Netflix/Akamai traffic, packets started dropping and Netflix customers on Comcast network experienced poor service. It’s not all that different from when the police shut off access to a major highway and send people onto local streets: Traffic backs up and everyone’s unhappy.

Luckily, building new roads on the Comcast network isn’t as labor-intensive as building out roads in real life. Even so, when it came to adding new ports for Level 3, Comcast took the opportunity to try to renegotiate peering contracts with Level 3. “These fights over peering can be like third-graders squabbling on the playground,” said Patrick Gilmore, chief architect, network infrastructure for Akamai, in an interview last summer.

While the fuss over the Comcast and Level 3 spat eventually died down after the two parties agreed to some form of give-and-take on getting new ports on the Comcast network, it highlighted a basic fact of life on today’s internet. In many cases, as the big get bigger, the internet’s core weakness isn’t technical; it’s the fact that the main players are now so large that they are in the midst of negotiating a new balance of power.

It’s an IP version of the treaties and shifting alliances in Europe in the late 1800s that then fell apart during the first World War. And while lives aren’t exactly at stake in these peering battles, the outcome of these fights might change the way the internet works, making it more expensive to build a business on the internet and allowing ISPs to become a new gatekeeper.

  1. Huittinen Massive Wednesday, June 19, 2013

    One of the problems is, there is no financial incentive for eyeball operator to deliver what they sell, quite the contrary, as you’re getting paid flat rate, the less you deliver, the more profit you make.

    And as competition pushes acceptable price constantly (what other in your life gets cheaper all the time, even against inflation) at the same time as your demand for the service increases (new services like NFLX attract new BW) the sustainability of the business becomes questionable.

    Some bigger fishes like FT or Comcast are large enough that they can tell content-providers ‘well you can’t access our eyeballs unless you pay us. Essentially indirectly billing their own users again via the content-provider, clearly ineffective way to do it.

    This also is anti-competitive to lesser providers, who don’t have the eyeball volume to extort money from content networks to offset the decreasing ARPU and increasing demand.

    If consumers would pay for what they consume, fair price, then operators would have financial incentive to produce the bought service, as the more they can deliver, the more money they get. It would also remove the indirect, ineffective additional billing via the content networks as well as level the playing field so that big and small eyeball operators have equal opportunity to compete for the customers.

    Share
  2. spixleatedlifeform Wednesday, June 19, 2013

    Recently finished reading “Captive Audience” by Susan Crawford Copyright 2013 and I never expected to see proof of her predictions so soon.

    SPLF

    Share
  3. We have the same story in France, with Free / Iliad fighting overtly with Google over YouYube traffic. (the CEO talk about it in interviews)
    So Free ADSL users can only watch YouTube with stutters, pause-and-load, low 480p quality… Vimeo is lightning fast, by comparison.

    It has been the case for months, and it seems there will be change only if the UE enacts some net-neutraly policy and then it’s passed into law. So years…

    Share
  4. When content suppliers deliver the traffic flow all the way to the destination city, demands for traffic balance (and financial compensation) are invalid.

    Asymmetric routing (hot potato) is common in the Internet. Traffic is handed off to the destination ISP at the closest interconnection. If there is a substantial traffic imbalance, this imposes a penalty on the provider who is transporting the heavy-use direction over long distances – they must upgrade their WAN capacity. This motivated the balanced traffic models, and calls for penalty fees when traffic was imbalanced.

    This financial model becomes artificial and invalid when there is no costly WAN infrastructure in the path. Today, in many cases, traffic originates in the same-city as the ultimate destination. This is due to decentralization of sources, notably through Content Distribution Networks. If a traffic imbalance exists in this scenario, the “receiving” carrier (last-mile operator) incurs no costly WAN build cost. They simply need to provide the MAN and last-mile capacity needed to make-good on the bandwidth commitments that they have made to their own direct customers (and for which the customers are ALREADY paying the carrier).

    When the source is bringing the traffic all the way to the destination city, the source carrier (or originator/CDN) is now incurring the bulk of the cost. It is inappropriate for the receiving carrier to demand compensation fees. Essentially, the originator has delivered the traffic all the way to the last-mile (eyeball owner) network. The last-mile operator is ALREADY being paid by their direct customers to deliver content to them. The last-mile operator should accept traffic from sources within the same city as the destination viewer WITHOUT demanding any compensation from the source. The last-mile network is ALREADY paid by its own customers to do this job.

    Share
    1. InSearchOfBalance Thursday, June 20, 2013

      In the days of vendor-maximum 10G LACP bundles and multiple 100G being used for cross-connects, hand waving that the peering edge (capex one needs to budget for on cycles that don’t match massive volume shifts) and metro are cheap compared to long-haul is disingenuous. Long-haul in the US is rather inexpensive now (try crossing the oceans for real $).

      I also disagree with the assertion that we exist in a happy world where all content sources and sinks are local to each other. For any given network, there will be massive variability and even asserting that “most” are local is bogus.

      Share
  5. Vertical integration is dead. RIP. The last mile guys are trying hard to counter an inefficient, balkanized bill and keep model that grew out of the private IP networks of old with 2-sided settlement models which are even more inefficient.

    The thing that’s wrong with two-sided settlements, in addition to being a double taking, is that they are being used to support inefficiently structured vertically integrated carriers whose operating stack is threatened at every layer, as evidenced by google fiber and 802.11′s tremendous performance/price advantage over incumbent service provider models.

    The article should point out what the cost of additional ports really are for last mile ISPs. (The answer: pennies, if that, per subscriber) The core WAN’s pricing (data centers, fiber nets, CDN’s, etc…) reflects competitive marginal cost. The last mile reflects monopoly average costs. The CDN’s would be more than happy to put their cache’s as close to the end customer (as in the case of Sonic/Netflix above, which is a layer 2-3 solution), only the monopolies know that doing so would reveal the farce and fiction behind their pricing models. Bandwidth in this country is 20-150x more expensive than it should be since we remonopolized the mid and last mile over the past 15 years.

    The article confuses worlds of absolutes and relatives. Horizontals and vertical boundaries. Lower, middle and upper layers. The two real issues here are a) open or equal access in layers 1-3, which I addressed above, and b) the need to confront and understand the limitations of bill and keep. The latter limits new service creation, perpetuates monopolies, does not scale, and most importantly prevents/limits centralized procurement (800, VPN, prepaid, etc…).

    Balanced settlements (whose transaction prices reflect marginal costs and scale into the billions/trillions of sessions/second) are highly stimulative and pave the way for free or subsidized access at the edge by the core. Balanced settlement exchanges in the middle layers can actually create virtuous push-pull between upper and lower layers and from network to network. They are the most efficient means to clear supply and demand across all layers and boundary points.

    The 2-sided models being promulgated by Mung Chiang at Princeton and numerous academics supported by the monopolies (and the cabal behind the ITU) is really a vestige of the inefficient, overpriced access and intercarrier compensation models of the ignorant regulatory regimes who have been duped by monopolies since 1913. It’s time they started learning about network theory from the 3 waves of voice, data and wireless digitization of the 1980s-90s.

    The issue highlighted in this article is really no different from the attempts by the Baby Bells to charge twice for data transport in the late 1980s and early 1990s after they unwittingly laid the economic foundation of the internet with their expanded flat-rate calling areas. The US dominated the internet because it had a 5-10 year head start with “free/unlimited” dial-up access to data networks. The internet was an accidental result of a monopoly pricing response to the competitive WAN/IXCs intelligent, metered, low-cost service model. The trojan horse was in the city and the downfall began. We’re just watching the final battle 20+ years later, the outcome of which is inevitable.

    Share
  6. There’s an easy way around this, if YouTube or Netflix would allow it. Just let people download the shows/videos in advance instead of requiring streaming. YouTube has recently cracked down on the Firefox plugin ‘Video DownloadHelper’, maybe they should change that policy. If a ‘content provider’ forbids it, viewer numbers for that content would drop. Too bad.

    Share
    1. InSearchOfBalance Thursday, June 20, 2013

      That falls down in the licensing distribution model (and for user-generated content, agreements the content providers have with their users). Also why we’re stuck with DRM.

      Share
  7. InSearchOfBalance Thursday, June 20, 2013

    Hyperbole is great for stirring up traffic – even detractors like myself have to come by just to pierce the bubble so this doesn’t get passed along as accurate or real news by the gullible. I’ll not comment on every line of breathless alarmist assertion and conjecture as that would take far too long and I’m sure there’s a longth limit to these posts. But here are key issues with a few choice nuggets:

    There is no “fierce battle”, there is only normal course of business.

    “Reports have been filtering in” … did someone here finally read DSLreports? While it has its utility, you can tune in any day of the week (for as long as the site existing – dig into the history) and find crowdsourced complaints across all service providers.

    “As they do so, they are taking agreements that used to be negotiated by engineers based on web traffic and changing them into disputes negotiated behind closed doors in boardrooms, executives fighting” – welcome to a decade ago. Please don’t go quoting the OECD report regarding handshakes, as they expressly were looking at exchanges and lower-on-the-long-tail than the large entities (many of which whom do nothing but private arrangements).

    This is not a “battle for the Internet” nor is it new. Settlement-based peering (not full transit) has been around for Quite Some Time. For example, many networks were built in the dot-com boom where the “settlement” took the form of fiber IRUs or telephony deals… which were so much more lucrative that “throwing in IP peering” made business sense.

    The uptick of settlement-based arrangements was at some point when AOL was building their global network and muscled their way into the Tier 1 club, and it was driven by up and coming networks seeking direct interconnections with the … ten? networks that made up that club at the time. Sadly, this was and is less a technical issues and more of an issue of irrational marketing cachet. Due to propaganda and ignorance, some network-service buyers took “transit free” as a proxy for “global, well-run, scalable” network. So your entire premise that “Big ISPs are killing things by forcing this business model on others” is actually backwards.

    Share
  8. alex simonelis Sunday, June 23, 2013

    Bottom line: big content suppliers like Netflix have to pay for delivering their product, and the payments have to be fairly distributed.

    Share
  9. The battle is not new, it has been going on for a while and it will get worst. They are simply moving to more creative ways to go to war against net neutrality when they were thwarted each time. I’m not sure if the telcos have a case against net neutrality yet because in spite of the internet’s growth ever since the ISP was introduced more than 20 years ago, their profitability seems on the up. They should realise that the business model has changed forever (analysed in my blog http://internetbusinessmodelasia.blogspot.sg/2013/06/reimagining-telco-impact-of-internet.html) ie. no basis to try to make the internet like the phone system. In fact the very idea is ludicrous but perhaps as one commentator said, “They (regulators) probably also suspect that some ISPs may be opportunistic”.

    Share
  10. Michael Elling Monday, August 19, 2013

    This is not a new battle. 100 years ago in the Kingsbury Commitment AT&T specifically wrote in 50 mile exclusion zones for interconnect because radio/microwave was already a known interconnect facilitator and/or bypass threat to AT&T’s WAN/MAN monopoly. Then in the 1980s the Baby Bells expanded LATAs and went to flat rate pricing to ward off the real competitive WAN, “metered” bypass threat from the IXCs to their Class 5 hegemond. Little did they realize they had opened the back door to data bypass from the nascent (and little understood) commercial ISPs with flat-rate dial up plans percieved of as “free” or unlimited use. The FCC kept this pricing back door open with Computer 2 in the early 1990s, otherwise we never would have had the internet (or at least the way it happened). We lost 1/2 of this equal or open access with the advent of vertically integrated broadband in the late 1990s and it was for all intents and purposes closed with Band-X.

    So this is not a new issue at all, but the heart of the battle between horizontal and vertical, open and closed, competition and monopoly, and digital and analog pricing models. Strictly speaking the only form of equal access is Wifi in layers 1-2 and offload from smartphones in layers 3-7 thanks to Steve Jobs.

    The net neutrality debate is actually not the real debate we should be having around open or equal access, but people think “neutral” peering or bill and keep exchanges is the economic engine at the heart of the internet success story. Frankly it is not. It was edge access pricing and commoditized WAN transport pricing that were the layer 1-2 commercial/economic foundations of the internet. And this was all a result of dial-1 equal access in long-distance. Neutral peering was possible because the marginal cost was so low and the volume of sessions/traffic didn’t warrant implementing metering.

    The world is a lot different now and we need to revisit bill and keep and look at it’s stifling aspects and instead focus on the very positive benefits of balanced or bilateral settlement systems in a mobile and HD video, 2-way world.

    Share

Comments have been disabled for this post