Blog Post

Peering problems aren’t technical issues, but economic ones

Congestion between last mile ISPs and other internet participants such as Cogent or Netflix isn’t the result of technical issues, but business ones, according to a report put out Tuesday by the Measurement Lab Consortium, a group of research, industry, and public interest partners housed at the Open Technology Institute. The report pointed out that these interconnection points — where different networks come together — are congested only in some instances and not in others. However, wherever such congestion occurs, it’s the end consumer who gets the shaft.

The M-Lab report covers peering, the practice where large network or content companies agree to share traffic between networks at various interconnection points. Companies can pay to peer with networks as Netflix did with ISPs, or in most cases, they agree to exchange their traffic freely sharing or having one party take on the minimal cost of servers and equipment needed to make a direct interconnection. Peering is different than buying additional capacity (called transit) on networks owned by companies such as [company]Cogent[/company], [company]XO Communications[/company] or [company]Level 3[/company]. Even major ISPs such as [company]Verizon[/company], [company]Comcast[/company] and [company]AT&T[/company] also offer transit capacity for a fee.

Screen Shot 2014-10-28 at 1.38.42 PM

What’s at stake in the peering debate

The report points out that such congestion occurs primarily between certain providers and during the “peak hours” of between 7pm and 11pm. This is what the FCC and others have defined as internet prime time. The report, which has been a long time coming, is part of a complicated effort to understand how the agreements between various networks that make up the internet affect the end user.

Earlier this year, we covered the issue in depth, including data from M-Labs, as part of [company]Netflix’s[/company] fight to avoid paying direct interconnection fees with the four large U.S. broadband providers. At the height of this fight between Netflix and the big ISPs consumers saw their video streams fail, because there was a bottleneck between Netflix and its transit providers where the Netflix streams hit the ISP network. The result was the user saw their bandwidth available for Netflix shrink to 1 Mbps or even less and their quality diminished. This even happened on YouTube and other video providers such as Hulu and Amazon.

Those issues looked something like this.
Those issues looked something like this.

It was a battle Netflix lost. It now pays Time Warner Cable, AT&T, Verizon and Comcast to peer directly, but it is still fighting the issue with the FCC. Indeed, the FCC is investigating the peering question and gathering data on the technical and business issues that play into the interconnection fights. I’m hoping the agency finds this report helpful as it tries to wade through ISP arguments that basically boil down to: Netflix is responsible for almost a third of the traffic on broadband networks in the U.S. so maybe it should shoulder some of the costs of providing that network.

It’s network neutrality all over again

That’s a fair argument to make, except it ignores that consumers already pay ISPs to provide that network, and the ISPs seem to be able to continue providing service and upgrades even as they continue profiting off their residential bandwidth services. The ISPs also ignore that they appear to be using their duopoly last-mile access to the end consumer as a means to extract payment from companies that want to reach them — something Netflix feels is a violation of network neutrality and even something at which the FCC looks askance (even though it is adamant that interconnection is not a network neutrality issue).

It’s also worth noting, as the Measurement Labs report does, that two providers — Cablevision and Cox — don’t see the degradation at interconnection points that the four largest providers do. Both of those companies accepted Netflix’s content caching boxes as part of its Open Connect service, where Netflix puts equipment inside the ISP’s data centers and delivers the content to it as a way of cutting down on the bandwidth required to deliver content and as a means of ensuring a higher-quality experience for Netflix customers.

Dave Schaeffer, the founder CEO of Cogent Communications and who has been incredibly vocal about ISP tactics in the peering fight, explained that there are eight ISPs worldwide exhibiting “cartel-like behavior” trying to get companies like his to pay for direct peering. The eight companies are Telefonica, France Telecom, Deutsch Telecom, Comcast, Time Warner Cable, CenturyLink, AT&T and Verizon.

Schaeffer said the problem hasn’t actually gone away even after Netflix signed its deal with ISPs. Cogent and others are still feeling pressure — a pressure that it has tried to address directly through negotiations with the ISPs and now through complaints with the FCC.

And that’s what’s most discouraging about the data in the report. It still shows degraded traffic patterns today between middle mile providers such as Level 3 and Cogent in certain areas for the country, even after Netflix capitulated on payments. This shows that Netflix isn’t the only target here, but any content provider paying for transit from middle mile companies. It’s likely some of the larger internet companies are getting hit up for direct paid peering agreements to bypass this manufactured congestion. Schaeffer says he is.

From the report:

[blockquote person=”” attribution=””]As in Atlanta, download throughput for Verizon customers in Chicago begins to decline in June 2013, and experiences the same minor and short-lived increase in March and May 2014. In Chicago, between September 22 2013 and March 2014, download throughput remained consistently less than 10 Mbps, with a low of 4.9 Mbps in February 2014. The performance change that occurred between March and May 2014 led to an increase in download throughput of 5.9 Mbps. Despite this, download throughput to the Chicago sites had yet to exceed 14 Mbps, and the most recent data suggests that download throughput has declined again.[/blockquote]

What’s the solution?

While this data is only that — data — and the report tries hard to stick to technical measurements, there are underlying economics and competitive issues at play here. For example, the cost of buying more servers for creating more interconnections between Cogent and Verizon’s network would cost about $10,000 per 10 gigabit port, said Schaeffer.

“For [all eight ISPs trying to charge for peering] to make problem go away, the one-time capital cost would be less than $10 million,” Schaeffer said. “We have gone as far as saying that for about the 20 percent of internet that travels over Cogent, we would pay 100 percent of our costs and the ISP costs to upgrade. We have pledged to any ISP that we will buy the equipment or pay them to make these upgrades, but even though these eight companies combined spend $100 billion a year in capital expenditures they can’t find a way to make a $10 million problem go away.”

If we can divorce the conversation from one about limited capacity because of technical reasons and cover the real issue — that some ISPs are using their monopoly access to the end consumer as a way to charge content companies money and control what goes on their pipes, we can have the discussion about peering that we need to have.

This report should help give the FCC some of the data it needs to move the conversation in that direction. For consumers or other interested parties, M-Labs also created a site called The Internet Observatory that lets you see how different connections between ISPs and other middle mile providers are handled around the country. M-Labs also makes it data available to all, so you can download it and see if you can come up with different metrics or tidbits about interconnection problems.

9 Responses to “Peering problems aren’t technical issues, but economic ones”

  1. Mother of Mercy, Tim! We do NOT agree! Why did you say that? To confuse people who are just skimming over the comments? No-one has taken the position that content providers should not have to pay their fair share. And there’s nothing special about the ‘last hop’ other than it’s the zone no-one can reach without the monopoly ISP’s blessing. There shouldn’t be a separate bill for that. Comcast has taken the silly position that their customers have only paid to REQUEST content, and not to receive it! Insanity!

    By your tortured reasoning, any content provider who doesn’t have a personal agreement with each and every ISP is “getting their bandwidth for free” from it. And that’s so far from the truth, it must be a deliberate deceit on your part!

    Content providers are paying their transit networks to carry their content onto the Internet, while ISP customers are paying to request and receive the content of their choice from the Internet. Content-server and end-customer may be paying varying rates, but the bill is split between them. Monopoly ISPs like Comcast aren’t paying anything, really. They maximize their profit through tricky maneuvers to hold onto as much of the revenue they collect as possible.

    They do this the traditional way, which is to charge more for the service than it costs them to provide it. But they also do this by lying about how big the burden is for them to transit data as they are contracted to do. They do this by guilting (and threatening) customers into using their service less. They do this by demanding more revenue. They do this by pretending their customers haven’t already paid them for the access to Netflix.

    Content providers have NOT always had to have individual agreements with ISPs to carry their content. You set up shop as an Internet Service Provider, and by definition you agree to carry whatever your customers request. Anything else makes you a tollkeeper. It makes you an obstacle that has to be paid to step aside and let the Internet work.

    As unscrupulous tollkeepers, they’ve placed more burdens in front of Netflix than other content providers. This is because Netflix competes admirably with their own video content services. Netflix now bears more than their fair share of the burden to bring the content to customers. And Comcast has to share that with NOONE now.

    The eye of suspicion is now firmly cast on any ‘mike’, ‘matt’ or ‘tim’ who pleads on behalf of the sacred right of an industry to screw it’s customers.

  2. This is much simpler than this long debate. If you believe that because you payed for Internet the businesses you visit online should not have to pay, than you will undermine the very infrastructure that you enjoy. Netflix is a for profit company. I connect businesses of all sizes from latte stands to fortune 50 companies to the internet everyday. If you think businesses have a right to free internet than you must be a socialist. Really, think about it. I would no longer have a job or would then work for the government, as would we all. Netflix and those on there side are pulling wool over your eyes to put more coin in their pocket. GREED.

    • Matt…. how do we put this… Nobody thinks Netflix should get their Internet for free. Stop parroting that. Obviously, Netflix is paying their upstream provider for access to the Internet, and ultimately to Comcast. Comcast’s customers are paying the other half of the bill by contracting with Comcast for Internet access to all things good on the Internet. Comcast is a for-profit company too, and they loathe sharing their revenue with their upstream providers. So they made up some cock and bull about paying “too much” to carry the traffic of one of their customer’s favorite service providers. Never mind that they’d lose a legion of customers if they didn’t carry Netflix.

      Netflix offered to provide a bypass at their own expense to save Comcast from the bandwidth bill from its own peers. This alone actually would have gotten Comcast some of it’s bandwidth for free. That wasn’t good enough for Comcast, who demanded additional payment from Netflix for the right to give Comcast free bandwidth. Get your head out of that hole. you must be a dunderhead to miss that. Netflix was offering to bear the full bill of their own traffic. Now Netflix is paying for the bandwidth AND paying a steep convenience fee to Comcast. Comcast is literally being paid to consume bandwidth now.

      The other thing is the ridiculous profit margin. This is how much it costs to provide, say, “a gig” to an average customer, including everything: —–

      This is how much the average customer is billed for each gig:
      —————————————————————————————————-

      And it isn’t to compensate for some minority of “abusers” who consume a thousand times the average amount of bandwidth.

      The greed is Comcast’s. The wool is knitted into that fine story of theirs that you seem to accept without question.

      The telcos have it set up so that restricting the bandwidth raises their revenue. No infrastructure upgrades to install, no relief for their video service competitors. It’s an embargo. Any expansion of the capacity means they must charge less for it, as higher consumption becomes the norm. Customer’s pockets aren’t getting any deeper so they can’t charge (much) more. They’re betting that they don’t have to keep up with demand as they are obligated to. Train customers to feel guilty so they use less! As long as they have a monopoly, it’s the cheapest and most obvious strategy to retaining their stranglehold on the industry. Lie about how much bandwidth costs. Americans don’t know anything about foreign news anyway. That $20/mo gigabit service in rural Romania must be some sort of complicated anomaly, eh? A lot of Matts get paid to install and maintain that modern network, so no boo-hooing about the stability of your jorb if USA monopelcoms are compelled to enter the 21st century.

  3. I still haven’t seen an article on peering, bandwidth prices and “shortages”, and network neutrality that wasn’t obviously made with the exclusion of a large portion of the whole picture. Sure, you can write an article that seems to say something intelligent and solid, by being selective about what you include and exclude. Most readers won’t know.

    You were far too kind to corporations in this article. Peering is an “economic issue” only in the sense that Internet service is a cash cow, and corporations are fighting dirty to keep milking it. These corporations are operating at 90-95% profit margins. This means that the high numbers presented as “costs” of providing service are artificial, and beneficial to these corporations as it gives them pretense for declaring one emergency or another (raising rates), and keeping smaller players out of the game.

    When two backbone providers make a “gentleman’s agreement” to charge each other, say, $100 per gigabyte, but keep peering close to 50/50, then neither of them has to pay the outrageous markup, but they can point to that as the market rate when independent ISPs complain. I do not see dirty tricks as being legitimate “economic” realities. Legal gymnastics to expand monopolistic opportunities are not “market realities”.

    As for Netflix, it should be emphasized that first, Comcast claimed that Netflix was costing them too much to carry their traffic (which is one of the services many of Comcast’s customers did business with Comcast in the first place to receive) over regular backbone connections. Then, Netflix generously offered to provide their own direct connections to Comcast, at Netflix’s expense. To which Comcast disingenuously screamed “Netflix is trying to blackmail us into giving them free bandwidth!”.

    Now, Netflix has been blackmailed into paying Comcast for the privilege of saving Comcast the expense of that bandwidth. In the long run, it might work out about even, since Netflix may be simply paying Comcast what it would have paid to it’s own backbone provider, but Comcast is still demanding double payment, and hijacking money that should benefit the Internet industry instead of keeping it all for themselves. Hell, Comcast may have pressured Netflix’s provider into jacking rates to force this “compromise”. Down this slippery slope, look for more big telcoms to bully popular service providers into paying for direct connections instead of using the public Internet. We don’t need more private commercial networks, we need faster public Internet!

  4. Wow, way to take a straight report and insert all of your personal biases trying to create a story. Oh, well, have come to expect this from GigOM article creators of late.

  5. What is so wrong with content providers footing more of the bill for ISP networks? The cost of transit has fallen from $1200/Mbps/month in 1998 to less than $1/Mbps/month in 2014. http://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
    How about letting those prices stabilize, or even go up a bit, in exchange for across the board rate cuts on consumer broadband service? Why are consumers so insistent that only they should be allowed to pay? Are they just being duped by the content provider PR campaign?

    • Well Tim, remember that as the cost of transiting content drops, the price of Internet to customers is still rising, so it seems like customers should either enjoy a discount, or the ISP could stop their silly pretense about these content providers costing them so much of their precious profits. What’s so wrong with ISPs sharing some of their revenue with their upstream providers instead of demanding that popular, yet economically vulnerable content providers bring them a free pipe, and pay a hefty convenience fee for good measure?

      • Yes, exactly, customers would enjoy a discount if more of the dramatic cost savings that content providers have been seeing were passed on to consumers. But, that can’t happen with this silly idea that transit networks and CDN’s should not pay ISP’s to complete the last hop delivery. That would leave the entire burden of funding the ISP network on the end consumer. I say “would” because content providers have always had to pay the big broadband ISP’s –Verizon, ATT, Comcast, TWC– to feed data into their network. So in fact they have been helping to pay for the ISP networks. Let’s have more of this, not try to outlaw it just because it is painful for Netflix.

  6. Steve Noble

    This is only a complex issue due to bureaucracy. The problem is the lack of a good solution. I personally dealt with peering spats with UUNet, Sprint, MCI and others back in the mid 90’s. The value of the eyeball vs the content is quite subjective and each side thinks it is correct.

    While there are ways to police the process such as requiring multiple POPs and IX connections from both sides, the ability to transport traffic across x (be it a continent, multiple continents or all) and other base requirements, there are no hard and fast rules that apply to all situations.