5 Comments

Summary:

Consumers complaining of poor Netflix and YouTube streams on certain ISP networks are the pawns in a fight over internet business models. Too bad knowing why this happens doesn’t fix this problem.

peering equinix carrier fiber cables
photo: Jordan Novet

On a message board under a thread titled, “Is Comcast blocking Netflix,” customers of the nation’s largest ISP are complaining about the poor quality of Netflix streaming during evening hours and wondering what’s going on. Their comments are both a reflection of Comcast’s previous bad behavior — blocking P2P content on its network back in 2008 — and evidence of frustration that they can’t get the content they want on the broadband network they pay for.

” I’ve got Extreme 105 plan and all speedtests show indeed about 90+M down and 10+ up. Netflix works too, but strictly in non-HD mode. When I force HD it starts pausing for buffering every 15-20mins for another 5 or so mins.”

I’m having major problems with Netflix. My option to stream anything in HD is gone. … Nothing has worked. When i attempt to watch a Netflix movie in HD on my laptop it does the same thing. Someone please help!

Waiting loading

This has been going on for weeks with a variety of complaints, troubleshooting attempts and arguments over network neutrality. There’s also a response from a Comcast engineer who explained that while Comcast manages its network it can’t be held responsible for the decisions of application providers and how they might route traffic.

What these consumers may not realize is that their frustration and complaints are actually a key element in negotiations between Comcast and the giants of the internet. It’s a negotiation playing out with greater frequency as many ISPs (not just Comcast) try to enforce their vision of how the internet should be paid for, and it’s a vision that internet content companies don’t share.

Several fronts in this fight

Another example of this fight can be seen earlier this week when Google launched a tool that measures the quality of a consumer’s internet connection. Called the Google Video Quality Report, the idea is simple: consumers can use the tool to measure how well their broadband handles streaming video. The service is launching in Canada first, and will presumably move south of the Canadian border in time.

Folks fond of testing their broadband might wonder why we need with yet another option for broadband speed and quality testing given that we have companies like Ookla, Speedof.me and even the U.S. Federal Communications Commission monitoring our broadband quality through a program administered by U.K. firm Sam Knows. There’s also a collaboration of application providers (including Google) and researchers called M-Labs that offers a broadband test.

The results of the Google Video Quality Report.

The results of the Google Video Quality Report.

But the Google Video Quality Report isn’t actually a broadband test. Like a similar service provided by Netflix, the Google report is designed to show consumers one thing — the quality level that Google’s YouTube videos could have when they get to the end user. When consumers see lower-quality video they naturally wonder what’s causing the problem, but asking that question puts YouTube watchers in the same position as those poor Comcast customers.

The consumer always loses

When it comes to delivering internet video there are many points of potential failure along the way to the home computer. But one likely point of failure has emerged as the Rubicon in the current impasse between ISPs and select content providers: the point where traffic enters an ISP network. This is a point where some ISPs believe they should be able to charge companies like Google or Netflix –which send the ISP a large amount of traffic — a fee for sending that traffic over the ISP’s network.

While this may sound like a network neutrality issue, it’s not. It’s a far more hidden and seemingly capricious problem: It’s a peering dispute. Peering governs how networks directly connect to each other on the internet, and most people like it because its an efficient way to exchange traffic across networks, so providers don’t have to build capacity to every single location in the world. And when it comes to peering disputes the customer always loses, because the customer becomes a pawn in the negotiations between internet superpowers (or wannabe superpowers).

When you’re bored at home and you request a YouTube video, it gets sent from Google’s servers over a variety of networks including Google’s, those owned by transit providers such as Level 3 or Tata and eventually your ISP’s network. (The internet is really a network of linked networks.) All along the way there are interconnection points where the video moves from one network to another. Ideally, it will take a path that is both short in distance and requires the least amount of hops from network to network. The closer the content to its destination and the fewer network hops is has to make, the cheaper it is for the person shipping the content and the end result is a better quality video.

But at some point the YouTube video will meet your ISP’s network. If the ISP peers with Google — and Google will peer with any ISP that can reach one of 70 Google points of presence around the world — then your video zips right through because both Google and the ISP have reached an agreement to make sure there’s an open traffic lane. But if an ISP doesn’t peer with Google then your video might get stuck in the equivalent of a digital traffic jam trying to get onto the ISP’s network. That means the user’s video might pixelate, stutter or drop down in quality.

goodvideo

Google and other firms that send of lot of traffic to end users are generally fans of open or settlement-free peering. In that scenario traffic zips around the web with everyone making sure the interconnection points have the capacity and servers they need for ever-growing traffic loads. But, there’s another model on the web — paid peering. This isn’t new, but it has been the source of some pretty well-known dramas online from fights between Cogent and Sprint back in 2008 to the 2010 drama between Comcast and Level 3/Netflix. Last summer a peering fight between Verizon and Cogent hit the news when Om Malik reported on the practice called out by Cogent’s CEO.

It’s a religious war

Companies that advocate for paid peering are arguing for parity in terms of traffic sent over their networks. If two networks both send equal amounts of traffic to each other, they do open or settlement-free peering. ISPs that own large backbone networks like Comcast or AT&T that cover the country (and who are the main proponents of this model) argue that they invested billions in building these networks, and that the traffic that traverses their networks creates costs for them, so other companies that send disproportionate amounts of traffic should pay to offset that investment and those costs.

The flip side to this argument, advanced by some transit providers, application providers and even the OECD is complex, but it boils down to three points:

  • It doesn’t cost that much to send bits over the network because ISPs buy capacity in bulk, not on a per bit basis. Plus big internet companies are willing to offset that cost with their own caches on an ISP network or building out their networks to meet large networks.
  • Because of a lack of competition in the U.S., ISPs are using peering as a way to extort money from companies like Google and Netflix so they can reach the ISP’s customers.
  • Imposing a mostly paid peering model on the internet will raises costs for everyone.

It’s also worth nothing that paid peering agreements are relatively rare, according to a 2012 survey from the OECD. That survey looked at 142,000 peering agreements, and found that the terms and conditions of the Internet interconnection model are so generally agreed upon that 99.5 percent of interconnection agreements are concluded without a written contract and without payment.

In short, it’s kind of like a debate between the business interests of individual companies versus the benefit of applying an unpaid model to the entire internet. Neither side wants to give in. For content companies, crossing that Rubicon means giving in and paying ISPs for delivering traffic they feel ISPs should be delivering as part of their compact with the end consumer. For ISPs, giving in means giving up one of the last control points they have if they hope to avoid becoming just a dumb pipe shunting bits from a content creator to the end customer.

Meanwhile, we as users just stare at charts from Google or Netflix showing that our ISP is behaving badly, while oftentimes being able to do little about it. And Google and Netflix have to hope that our pain as consumers will force the ISPs to change their mind — or regulators to get involved.

These are high stakes negotiations, and its likely we’ll see more of them as time goes on.

  1. Sure, and Amazon should pay UPS because of all the airplanes they have to buy. Basically, the ISPs want to charge both sides, the content providers and the consumers. The crazy thing is that usually companies that generate a lot of business for distributors get better shipping rates (e.g. Amazon and UPS). Netflix and Youtube are creating demand for the ISP’s services. It’s nuts to gouge them. Of course, the real difference is a lack of competition. UPS still has competition but ISPs don’t anymore.

    Reply Share
    1. Great point McLars!

      Reply Share
  2. “argue that they invested billions in building these networks” …which they received tax subsidies to build so that taxpayer’s paid for it all out of their pockets and not the companies.

    Reply Share
  3. Robert M. Enger Friday, January 24, 2014

    It is worth noting that many content providers use intelligent server selection algorithms. These ensure that the content that the end-user requests is delivered from a server that is located as close as possible to the requester. This minimizes burden on the wide area (backbone) networks.

    In many cases, content is delivered from servers IN THE SAME CITY as the requesting end-user. But, the content-servers and the requesting end-user are connected to different ISPs. A common example from your story is: content-server connected to Cogent, end-user connected to Verizon. Even through the content originates a few miles away from the end-user, in the same city, Verizon still refuses to allow the content onto its network unfettered. Clearly, there is no cost-burden on Verizon’s WAN backbone. The traffic need only traverse Verizon’s in-city and last-mile network facilities.

    The end-user pays a (often substantial) monthly fee to Verizon. If this fee doesn’t guarantee proper use of the in-city and last-mile network, what exactly is the end-user purchasing from Verizon Internet? An empty promise?

    Reply Share
  4. Aidian Holder Monday, January 27, 2014

    Yeah, the ISPs should definetly get paid for delivering all those bits from the content companies. Lucky for them, they already are. I pay them every month. And as for dumb pipe, that’s exactly what I want, and exactly what I pay for (and exactly what they got taxpayer subsidies to be)

    Reply Share