51 Comments

Summary:

Verizon is locked in a head-butting battle with Cogent Communications, a large bandwidth provider. The cause for these issues: Netflix, one of Internet’s killer applications that has been growing its share of the network. Bad news for Verizon customers: Netflix may not work as well.

If you are trying to get Netflix and use Verizon’s broadband, then there is a good chance that your video performance is less than optimal. Some Verizon customers might even go as far as calling it a crappy Netflix experience. The reason: a behind-the-scenes power play between Verizon and Cogent Communications , one of the largest bandwidth providers. The head-butting between these two companies is over an arcane concept known as peering.

reed-hastings-happy-oPeering is essentially an arrangement between two bandwidth providers where they send and receive traffic from each other for free. The logic is that the data sent from one network to another is reciprocated. Verizon runs one of the largest last mile networks and owns the descendants of MCI. Cogent is one of the largest bandwidth providers, and its network is spread across the globe in hundreds of cities.

Cogent and Verizon peer to each other at about ten locations and they exchange traffic through several ports. These ports typically send and receive data at speeds of around 10 gigabit per second. When the ports start to fill up (usually at 50 percent of their capacity), the internet companies add more ports. In this case, through, Verizon is allowing the ports that connect to Cogent to get crammed. “They are allowing the peer connections to degrade,” said Dave Schaeffer, chief executive officer of Cogent said in an interview. “Today some of the ports are at 100 percent capacity.”

“Think of it as the on-ramp to the freeway being log-jammed,” Schaeffer said. And that means your Netflix content, especially content sent by Netflix’s content delivery network, slows down, and you get pixelated pictures and buffering.

While not naming Netflix directly, Verizon has indicated to Cogent that the reason behind its actions is that Cogent is moving traffic for a large video provider. Schaeffer confirmed the Netflix is one of their largest customers. “Over the past year Netflix has become a big partner for us. This is a business model problem, not an engineering problem,” Schaeffer said.

Our sources tell us that Netflix recently bought 2 Terabits of bandwidth capacity in part to get around such cramming that was happening in places where it sends traffic directly to certain internet service providers.

When we called Verizon about this story asking if Verizon was having a problem with Cogent over peering issues associated with Netflix, Verizon spokesman Bill Kula said he’d get back to us. A few minutes later he sent the following reply that didn’t answer our question:

Verizon operates one of America’s lowest-latency, highest capacity networks. The various classes of Internet speeds we offer are among the fastest in the nation. Time and again, customers rate us best in class in various reports and surveys. Our customers enjoy a consistently superior Internet experience because our networks can adapt and grow with their use.

Netflix has been growing like crazy and it now accounts for a whole lot of Internet traffic — almost one out of every 3 bits (32.3 percent) sent downstream to users in North America is Netflix traffic according to Sandvine, a company that makes traffic monitoring gear for ISPs. That’s a lot of congested ports.

Netflix’s growing popularity has made it a target of ISP (internet service providers) vitriol and anger, especially those who offer competitive services. Verizon, for instance owns 50 percent of Redbox, a video-over-the-Internet service that is competitive with Netflix. Time Warner Cable and Comcast are other large providers that has allowed degradation of the online video experience on its networks — after all the logic is that as people start to have a bad Netflix experience, they start to look for alternatives — perhaps the ISP’s own pay TV offering.

This isn’t the first application last mile network operators have tried to degrade — last year the wrath of the Baby Bells and cable companies fell on Megaupload, a file sharing company started by Kim Dotcom, Schaeffer said. That too was one of the big bandwidth-hungry services popular with the end customers of the ISPs — actual consumers.

This post was updated on Wednesday June 19th to correct the spelling of Dave Schaeffer’s name.

You’re subscribed! If you like, you can update your settings

  1. Just reinforces why Verizon is among the most hated companies in America.

  2. Richard Bennett Monday, June 17, 2013

    Why is it that every single Internet peering dispute seems to involve Cogent?

    1. Maybe because they are big and are offering their bandwidth at a really low price that doesn’t fit into the plans of the carriers and ISPs who want to keep an artificial premium? Or that would be guess at least.

      1. The Wikipedia page for Peering ( https://en.wikipedia.org/wiki/Peering#Depeering ) lists these “notable” disputes:

        BBN Planet vs Exodus Communications[6]
        PSINet vs Cable & Wireless[7]
        AOL Transit Data Network (ATDN) vs Cogent Communications[8]
        Teleglobe vs Cogent Communications[citation needed]
        France Telecom vs Cogent Communications[9]
        France Telecom (Wanadoo) vs Proxad (Free)[10]
        Level 3 Communications vs XO Communications[citation needed]
        Level 3 Communications vs Cogent Communications[11]
        Telecom/Telefónica/Impsat/Prima vs CABASE (Argentina) [12]
        Cogent Communications vs TeliaSonera[13]
        Sprint-Nextel vs Cogent Communications[14]

        For one reason or another, Cogent is more likely than not to be involved in these things.

      2. Corrected this for you:

        Maybe because they are big and are offering their peers bandwidth at a really low price that doesn’t fit into the plans of the carriers and ISPs who want to carry it more than 1 router card? Or that would be guess at least.

        1. it would be better Peerguy if you wrote out your explanation as a separate comment that we could learn from IMHO.

          1. Cogent (and a few other low-ball providers) are famous for selling 1-hop transit.

            This is when you go to a big content owner saying you can sell transit for well below anyone’s cost as long as that content provider connects to you in Site A on Router B and delivers to ONLY Customer 1, 2 and 3 which happen to be peering on Site A in Router B and many times on the same Nx10G card.

            Multiply that by 10 cities and a similar configuration which never requires any real network from the 1-hop transit supplier, combine that with overselling and congesting the existing capacity and you have a peering dispute

            1. Thanks you PeerGuy

            2. No problem. Education is needed around this topic. Chris Dickens below also clarifies the situation a bit further around the economics of peering and how it changes when someone abuses the relationship then publicly demand entitlements.

              Now that you know more, perhaps you can correct your title around who’s actually in control of this issue and more directly to “blame” for issues.

            3. I am going to stick to my headline for now. :-)

              PS: What is your affiliation @peerguy?

            4. InSerachOfBalance Peerguy Wednesday, June 19, 2013

              @peerguy – controversy sells papers^Wpage views.

            5. I’m no fan of either Verizon or Cogent, but Peerguy nails it. We run into this problem all the time. It’s called “peering”. It should be between peers. If one side it dumping massive amounts of traffic onto your network for free, there’s a problem there. Cogent’s customers are paying them for bandwidth, not excuses. If Cogent’s free peering and begging for entitlements is not working for them, they should man up and purchase bandwidth. I’m sure Verizon would be happy to provision a circuit to Cogent at market rates. Again, no fan of Verizon, but Verizon’s not the bad guy here and they’re not doing anything nefarious at least in this case.

              If you want to report the truth, have Cogent tell you on those points where they’re saturated (I know for a fact Chicago is one of them) which direction are the ports saturated. It’s a pipe. As you report, 10gb in each direction. Is the traffic load even, meaning that Cogent traffic is taking 5+gb AND the Verizon traffic is 5+GB, or is it one way, e.g. Cogent traffic is 5+gb, but Verizon is well under 50% capacity. I know (at least in Chicago) it’s all Cogent dumping traffic on Verizon.

    2. Not all of them, just a lot of them. Why? Well they obviously can’t keep their traffic ratios in line. They should work on getting more eyeball traffic.

  3. Verizon, Time Warner Cable and Comcast are all monopolies behaving in anticonsumer anticompetive ways.

    1. Not true – as VOIPCEO said, it’s usually an unbalanced traffic pattern. I know with TWC and XO, XO was dumping approx 200G of traffic on TWCs network, while only 30-40G of traffic was being pushed onto XO’s. The majority of the traffic was and is Netflix related….After additional contract negotiations with XO, things are working well again….The disparity of traffic loading and balancing has to be managed – it costs a shitton of money to support this traffic.

  4. Verizon’s peering is hot to a number of other carriers, including NTT. They are also doing some pretty scenic routing to Comcast.

    Why aren’t we looking towards Verizon, and not Cogent, as the problem here?

    1. David

      The problem is that Cogent doesn’t have a competitive product to Netflix like Verizon and TW Cable do and the fact that Verizon has been nudging them to shut down their client as the story reports.

      1. That’s an accusation without factual basis – how about some actual journalism and find a decision memo where a VZ exec says to run this peer specifically hot? Everyone in the industry knows that VZ bought UU and then handed the policy keys over to the people who wish to maintain the legacy “Tier 1″ structure. They run hot to those who don’t pay them. Occam’s razor says this is just another example. Maybe the hype around Netflix will make it the example that helps to shatter the behavior, but to pretend that it is more than just consistent behavior demands proof, not conjecture.

        Bring it.

  5. Would love to use Netflix streaming, but we have 3Mpbs DSL here, from Frontier. A struggle to watch anything, because of such slow speeds.

    I wonder how much of the USA is in the same boat, where Netflix streaming is just not practical to use.

    1. Illinoi

      I think the 3 Mbps is enough to watch Netflix on your iPad screen but I agree, you rarely get 3 Mbps and the networks are completely degraded.

      1. One of the challenges is that the revenue model for service providers is getting upside-down. The all-we-can-eat bandwidth at a particular speed enables applications like Netflix, but requires investment to support this. Netflix reaps the rewards on the back of service provider networks.

        1. ^ what you said….I wish more people understood this.

          1. I agree that Netflix is getting a great deal, but its to the benefit of the ISP.

            ISP’s charge inflated prices for modest speeds which we are willing because bandwidth hogs like Netflix are so popular. Without these companies there would be no demand for $70 internet plans.

            If I subscribe to higher tier plan, but I can only utilize a small portion of it due to congestion, what is the incentive to stay at that higher priced plan? Are they not shooting themselves in the foot?

  6. Chris Dickens Tuesday, June 18, 2013

    It’s not Verizon’s responsibility to provide Cogent with bandwidth. There’s nothing “arcane” about Internet peering. It’s quite simple. When the balance of traffic between Cogent and Verizon are similar then the links can be provided at minimal to no cost to each other. If there becomes a bandwidth deficit (lop-sided, think like trade deficits between countries) then the peer pushing traffic much harder is forced to pay a fair market value to increase the capacity of the connection in the direction their traffic is going and the agreement switches to a “transit” agreement.

    So, there are only one applicable question that needs to be asked before pointing a finger in the direction of Net Neutrality:

    Is Verizon charging an unfair amount or even denying Cogent a transit agreement? (Unfair as meaning dissimilar to what Verizon charges other transit customers — not the amount Cogent thinks they should be charged)

    1. Just to clarify the relationship does NOT have to switch to a “transit” agreement, it switches from a “settlement-free peer” read as free, to a “settled peer”. The routes exchanged stay identical, but the network sending (pushing) more traffic would pay the network who is receiving (transporting) said traffic.

      Transit would indicate that the paying party would have their routes advertised to the peers non-paid customers and that wouldn’t be in pushing parties best interest. The majority of “peering relationships”, between ultra-heavy content providers and ultra-heavy eyeball networks are based on a settlement basis, just not typically disclosed. Think of the Level(3) / Comcast dispute.

      1. cgucker – Thanks for clarification. I was unaware of the terminology of “settlement-free” versus “settled peer”. My input here is actually a direct result of lots of research I did back when the Level3 Comcast spat occurred some time back. I ultimately sided that as long as Comcast is not unfairly charging for the bandwidth versus other big data customers then they weren’t in the wrong. Similar situation applies here. If they are able to settle the peering agreement without providing transit (no additional routes) then the agreement may possibly be done more inexpensively than a transit agreement.

        The pickle is ensuring that Verizon doesn’t play favorites and charging reduced rates to other settled peers where the incoming traffic doesn’t compete with their own products. I’m not a proponent of government oversight, but some very limited eye-balling can go a long way to keeping everyone honest and fair.

    2. Verizon only has eyeballs. all content comes from Cogent, level 3, etc. if Verizon cared about their customers experience using the internet they would upgrade their capacity. that is how the internet was designed, to exchange information from one peer to the next. it is obvious that Verizon has an agenda.

      1. “Verizon only has eyeballs” is false. Loads of enterprise, direct on-net hosting, and downstream datacenters/hosting providers are on their network because of the UUNet acquisition. All the corporate entities have agenda: maximize profit, minimize cost.

        The rest, as with this supposed “reporting”, is conjecture. Pass the tinfoil…

    3. Actually, by definition, it is arcane.

      known or knowable only to the initiate : secret ; broadly : mysterious, obscure

      If the peer pushing is the one that pays, then I want the ISP I own to peer with Verizon and have them pay me to take their traffic. After all, they’re the ones pushing it to me.

  7. Mostly we have the alternative of dropping Verizon and such like companies and changing to a company that doesn’t throttle. However, who’s to know which alternative ISP doesn’t use Verizon? In some cases there is no alternative. This opens the question of whether to use Government lobby to legislate anti competition law. Will this interfere with free trade? A whole new bees nest.

  8. Most of us have the option of changing from companies that do this. Unfortunately there are some that have no option other than to lobby government for legislative changes to stop this practice.

  9. Brian Richards Tuesday, June 18, 2013

    Is this legal if they have a competing service?

  10. William Donelson Tuesday, June 18, 2013

    God forbid American companies should invest in America. T
    But the tax breaks we give the rich are not invested here, so why should the big companies do better…

Comments have been disabled for this post