The conventional wisdom is that Comcast is evil, therefore, Level 3 must be the innocent victim of Comcast’s capricious greed. In reality, this is a complex situation without clear-cut heroes or villains — in the network game, this is business as usual.

toll image

Let me offer a different, and perhaps, unpopular take on the current Comcast/Level 3 imbroglio. The conventional wisdom amongst Internet users is that Comcast is evil; therefore, Level 3 must be the innocent victim of Comcast’s capricious greed. I’m the first person to point out some real problems with Comcast; its secretive take on “network management” and ham-handed response to the regulators has made it no friends.  However, this is a complex situation without clear-cut heroes or villains — in the network game, the Comcast/Level 3 conflict is business as usual.

First, let’s dismiss the alarmism . No one is “breaking the Internet.” That’s sort of the first rule among network engineers and peering managers, and a great deal of effort goes into meeting this goal. (Stacey did a good job explaining the issues involved in her post.) A peering dispute of this sort can lead to a partition — a condition where some Internet users can’t reach others — but that’s rare and considered to be highly reckless by the network operations community. While Cogent has been willing to prosecute its goals by partition, most other providers find it to be anathema.

So, what happened between Comcast and Level 3? Prior to this dustup, Level 3 was one of Comcast’s Internet transit providers. Level 3 sold wholesale Internet capacity to Comcast (as does Tata). Comcast engages in settlement-free peering with many networks, but as it’s not 100-percent peered, it needs upstream transit providers, such as Level3 or Tata , to offer a route for all non-peered traffic.

Level 3 has recently become one (but not the only, by any means) content delivery network for Netflix, which has radically increased the outbound traffic from Level 3 to Comcast, threatening to saturate the transit links between the two providers. That’s just business: no harm, no foul. But what Comcast found troubling was that Level 3 now looked a lot less like an upstream transit provider, and a lot more like a content delivery network (CDN). This is an issue, because Akamai and Limelight pay Comcast to deliver their traffic to their end users. Level 3, on the other hand, is paid by Comcast, ostensibly for transit, but now, seemingly, to deliver traffic that Level 3 has already been paid for — by Netflix. Level 3 wanted to get paid both ways: coming and going. That’s nice work if you can get it.

Level 3’s transformation into a CDN raised additional questions. The era of the Internet backbone providers has been coming to a close. The Level 3s, Global Crossings, and such are in some ways relics; the real power is with the last-mile carriers — cable, DSL, FTTP providers — and the content providers  — Google, Microsoft, Akamai, Limelight, et al. These eyeball and content providers have been increasingly disintermediating the classical Internet backbone providers out of the equation by interconnecting amongst themselves. While providers like Level 3 still have some utility there, that utility is shrinking rapidly.

Comcast and its EVP of Networks, John Schanz, have surely been asking themselves, “Why have we been paying Level 3 for transit?” After all, Comcast is a much larger and more important network, especially in terms of traffic, in the general scheme of things. While many of the old backbones have been acquired — and more will be in the coming year — Level 3’s titanic debt load has scared off just about everyone, leaving it high and dry.

All of this said, no one here did anything wrong; these disputes aren’t uncommon. Well, no one did anything wrong until Level 3 whined to the press and regulators. When you’re on the losing side of a peering dispute, you’re always tempted to complain to regulators about how you’ve been treated unfairly. Then you remember a regulated peering arrangement is the only thing worse than what we have now. More regulation puts more power in the hands of the big providers, not less, as they have the experience, resources, and the legions of regulatory lawyers necessary to “play the game.”

Look at the Telecom Act of 1996; the Baby Bells won that fight due to their superior regulatory game-playing ability. The cable companies, PTTs (national telephone companies), AT&T, and Verizon would surely come out on top of any major movement towards peering regulation which would cut out the large number of small firms that peer 50 percent to 70 percent of their traffic.

So, how does this play with network neutrality? The short answer is that it’s completely unrelated. Netflix isn’t paying a dime to Comcast. Level 3, which sells services to Comcast, is. It’s the price of doing business when you have a large amount of outbound traffic. No user will pay more to watch Netflix movies. Netflix certainly won’t see this passed on to them; the low cost of Level 3’s service is what won the deal, and if that goes away, so will Netflix, to greener pastures. Instead, this is a battle where a provider that does no business with consumers (Level 3) will pay money to a company that does provide services to consumers (Comcast). There will be no prioritization of traffic as part of this deal, only delivery.

This in no way absolves Comcast of its duty — as good corporate citizens, not to mention a company trying to make a merger happen — to be more transparent, to deal openly with its customers, to improve its levels of service, to provide competitive prices. These are all areas where Comcast has a lot of work to do, to put it mildly. However, in this latest peering conflict, one of the dozens that I’ve observed or participated in, there is nothing malevolent or damaging to consumers, on either party’s behalf.

Daniel Golding is Managing Director of DH Capital, a boutique investment firm in the Internet infrastructure and datacenter sector. Prior to that, he worked at Tier1 Research, was the Global Peering Manager at AOL. Daniel is an engineer with 20 years of network and mission critical datacenter experience.  Disclosure: He takes no money from CDNs, Netflix, Comcast, Level3, or anyone else involved in this story, but in the past, he has sold research to those companies.

Image courtesy of Flickr user TheTruthAbout.

Related GigaOM Pro Content (sub req’d):

You’re subscribed! If you like, you can update your settings

  1. Comcast, on the other hand, is paid by me, ostensibly for transit, but now, seemingly, to deliver traffic that Comcast has already been paid for — by Akamai. Comcast wanted to get paid both ways: coming and going. That’s nice work if you can get it.

    Basically, this comes down to money. The only reason I’ll side with Level 3 is so that Comcast gets less money.

    1. Is Comcast asking L(3) to pay them? No. Did they ask L(3) for SFP and get denied? Probably. So Comcast is exerting pressure where it can, on its consumers’ and Netflix’s bottom line, which in turn motivates L(3) to act due to the potential loss of revenue. L(3) should give Comcast SFP, Comcast should roll back the subscriber premiums and this whole thing will go away.

  2. I agree with the gist of this. And I think Level 3 overplayed its hand.

    But there is a story here — and you touch upon it. The real story is that Comcast has grown big enough to dictate terms to the other Tier 1s. In other words, the story here is that the big end-user ISPs can now leverage their terminating monopolies in new ways b/c of their size.

    That’s a big deal — and it has a lot of implications. So even if Level 3 is full of it, the larger dynamic still holds.

    And it’s just genuinely dangerous to have end-user ISPs that are this big (and have this much leverage). It’s the shift in power to Comcast that makes regulation necessary — not necessarily this particular dispute.

    1. End-user ISPs “this big” have always been the case…look at AT&T and Verizon.

      1. You guys are both right. There has been a real shift from traditional “Tier 1″ networks towards the broadband carriers. Some of that shift happened due to acquisition – Verizon acquired MCI/Worldcom/UUNet (ASN 701/2/3). AT&T has always had a big backbone and acquired SBC and Bellsouth.

        The power shifts with the traffic. Luckily, there are enough players to keep things from becoming too centralized. Any further consolidation of major broadband providers could be rather dangerous.

    2. Richard Steenbergen Joe Wednesday, December 1, 2010

      The true power of Comcast isn’t in the size or scope of it’s network, it’s in the captivity of it’s customer base.

      If Level 3 turned off Comcast for refusing to pay their contractually obligated transit bills, the traffic would be forced through massively congested Tata transit ports, and a huge number of Level 3’s customers would take their business elsewhere as a result. If Comcast intentionally congests its transit providers and provides terrible service to its end users, which it has been doing for several months now, most of those users have no real alternatives to switch to.

      In other words, content is mobile, eyeballs are not. Comcast realizes that they can (ab)use their captive eyeballs to force content to pay them for access, without having to create a “100% down” partition like Cogent has done in the past. For a Netflix customer, 20% packet loss is effectively just as down as a hard partition.

      The irony here is that only a few years ago Comcast was the victim of this same maneuver, back when they were a single homed “captive” customer of AT&T. AT&T intentionally congested their peers by refusing to perform necessary upgrades (citing “poor ratios”, even though THEY were inbound heavy, and it would have been impossible for any balanced network to be truly balanced to them), to try and force content networks to buy from them directly if they wanted uncongested access to Comcast. Much like in real life, the victim has now become the abuser.

      Unfortunately I have to disagree with Dan on a few points. Specifically, while it should be every network’s right to choose who they do and don’t peer with, or buy transit from, things start to get murky when one network is abusing their franchise agreements and near monopoly or duopoly status in many markets. If users had an actual choice, and could get comparable broadband access elsewhere, then Comcast would be free to congest their network however they see fit. But that isn’t the case, and this is where government involvement and Net Neutrality start to have legitimate grievances with Comcast’s actions.

      But let’s also not forget the real outcome of AT&T’s actions a few years ago. Comcast, their largest customer, was forced to leave them and build their own backbone. They may have seen some temporary market success from people who were willing to pay a few bucks to get uncongested access, but it also hurt AT&T’s reputation as a reliable provider a lot more. In the end, all consumers want to buy from the best network offering the best service at the best price, and “networking extortion” through restricted access to your customer base will eventually do more harm than good.

      1. Richard, in the interest of full disclosure, is there anything about nLayer’s relationship with Comcast that you would like to share?

      2. I don’t think there’s much merit in trying to smear Mr. Steenbergen. All of the people with deep, current, and up-to-date knowledge of Internet operations are working for someone in the field, so this assassination game can be played any time.

        It’s much better – and makes you look less bad – to stick to the issues instead of attacking the messengers. Steenbergen raises some interesting points that deserve an honest response.

      3. My only relationship with Comcast is that it gets harder and harder to successfully deliver to their customers with each passing day. Honestly I don’t really care who Comcast does or does not peer with (as I’m sure Dan knows, the cost of transit these days very closely parallels the cost of peering anyways :P), but I do expect them to be able to take the bits somewhere. The real problem is that Comcast has restricted the capacity into their network, both via their transit and their peering, so tightly that it is nearly impossible to deliver the bits WITHOUT having to pay them for it. I deeply object to this kind of manipulation, especially when they’re using their monopoly/duopoly status in MANY markets to accomplish it, and I will not willingly be a party to it.

        I’m sure Dan knows all of this too, he just objects to government involvement in making Comcast play nicely with the other children, since they know next to nothing about Internet peering and could easily be lead astray by industry lobbyists. And I’m by no means saying that government involvement is the right answer either, but it is VERY clear that Comcast is abusing their position to degrade quality, limit consumer AND content provider choice, and force higher prices for everyone. Something needs to be done, and I for one applaud Level 3 for taking the risk to come out publicly and say something.

  3. I agree with most of this article, but to say “that it’s completely unrelated” to network neutrality is a bit disingenuous because a few sentences later you say:

    “There will be no prioritization of traffic as part of this deal, only delivery.”

    The problem is there could be. The decision to include prioritization as a term of any deal is at the sole discretion of the two parties. No regulation exists to give the consumer a voice in that deal, to protect consumers from being on the short end of the deal in a market where there are few if any alternatives.

    1. Why get the government involved in trying to prevent something that *maybe* *potentially* could ever happen?

      Technically Comcast could turn all your video a nice shade of blue, but no one is out trying to get laws past to stop the delivery of blue-toned video. Why? Because blue video would be bad for business. So would heavy-handed prioritization. So neither is going to happen anyway.

      Stop with the paranoia.

  4. Great article.

    A different view: Level 3 can’t play the CDN when it’s favorable to them (trying to get the multiples of the sector for their stock), without expecting that there’s a price to be paid (dealing with their partners in a different way).

    P.S.: for the editorial staff: kindly correct Dan’s name on top: By Daniel Golding

    1. Stacey Higginbotham Paolo Wednesday, December 1, 2010

      Paolo, fixed. Thank you for pointing it out.

  5. Although I think Level 3 is lashing out before its fall into obscurity I think the bigger issue is that comcast is trying to be both a transit service and CDN. When they are so big and have so much power they can charge fees and make whatever restrictions they want to these other CDNs and push them out of business (or purchase them).

    Once they own the system from start to end and the market continues to consolidate this will lead to the end of network neutrality.

    1. Richard Steenbergen Sean Wednesday, December 1, 2010

      I don’t think that’s true… Comcast has easily one of the most incomplete, poorly engineered, highest latency backbone IP networks I have ever seen. For example, it takes them over 109ms RTT to move a packet between San Jose CA and New York NY, something most other networks can manage to do in well under 70ms. Comcast doesn’t even BEGIN to compete in terms of delivering a product that is comparable to Level 3, their network is so poor that it is literally ONLY acceptable to the lowest end “ghetto hosters”, and home users who simply have no other choice in the matter. Comcast also lacks any kind of non-US presence, and has at best metro-level fiber footprints plus a very incomplete longhaul network they’ve purchased from Level 3.

      To claim that Level 3 will be “replaced” by Comcast is quite honestly absurd. The only reason people buy any kind of wholesale service from Comcast is to get uncongested access to their captive cable customers, and nothing else. They offer nothing else of value, in terms of either price, quality, or availability.

      1. I agree with some of what RAS has said – Level3 has a wonderfully engineered network. But from a business point of view, they are a disaster. Comcast is of interest to hosting providers not because of network quality but simply because of access to eyeballs. Comcast’s has only really professionalized their network team in the past few years. It will be 3 to 5 more years until you see a top-flight network there.

        That being said, I don’t think Comcast has any desire at all to be a CDN or to own “end to end”.

      2. Gee, Daniel, have you heard about Comcast’s proposed acquisition of NBC Universal? It’s been debated at length here on GigaOm, I suggest you bone up on it. If the merger is approved – and some suggest this dispute is an attempt to derail it – Comcast would become a major distributor of content across the Internet. And even if it’s not approved, Comcast already offers a “TV anywhere” package that streams TV shows outside their network.

        These developments suggest deep involvement in content streaming by Comcast.

  6. With this action, Comcast is preventing competing content from ever being delivered to Comcast’s subscribers at all, unless Comcast’s unilaterally-determined toll is paid – even though Comcast’s subscribers requested the content. With this action, Comcast demonstrates the risk of a ‘closed’ Internet, where a retail broadband Internet access provider decides whether and how their subscribers interact with content.

    why did aol use level(3) for its L2TP ?
    the world rides on Level(3)

    1. I like this model better than having Comcast silently QOS my data. As a consumer I know what Comcast is charging me extra for. If I am in a metro with options, this motivates me to reconsider who I buy residential data from and/or how important Netflix is to me.

  7. Glad to see some sanity here. The so-called “tier 1s” have been irrelevant for a long time now, lurching zombies propped up by contract interia and marketing FUD. If anything, their cartel of interdependence has proved to be a weakness in previous peering disputes, with no path of last resort.

    1. Yes, there was an article about all that on GigaOm a year ago that you commented on quite negatively.

      1. Richard – I commented negatively on the interpretation of the specific NPRM, which wound up being in line with Katz’s and Peha’s comments at NANOG (http://www.nanog.org/meetings/nanog48/presentations/Monday/net_neutrl.wmv – near the end). Unfortunately, looks like L3 seized on this relatively weak concept as a sabre-rattling technique but were called on it. I’ve *always* been against the bogus marketing of “tiers”.

      2. Right, Katz and Peha told the transit folk that the FCC had no intention to apply the proposed regulations in de-peering disputes, and you were satisfied.

        In the real world, however, the FCC’s Wireline Competition Bureau chief called Comcast as soon as this story broke. Comcast was then forced to explain why their actions didn’t violate net neutrality; and they’re still explaining.

  8. Director of AJAX Wednesday, December 1, 2010

    You stated “AT&T has always had a big backbone and acquired SBC and Bellsouth.”

    I think you know better that SBC acquired AT&T, which later then acquired BellSouth.

    Director of AJAX

    1. You are entirely correct. Same impact, though. Which set of drones consumed the other didn’t end up mattering. The SBC people were seemingly more AT&T than AT&T.

  9. Daniel
    Thoughtful, and I happen to know this is how things started out in the dispute. But ultimately I think it’s not a good thing for the Internet.

    Follow the money.

    Internet traffic is currently about 4:1 download:upload and the ratio is increasing. That means virtually no backbone can be symmetric with any large broadband provider and by Comcast’s interpretation must pay Comcast for the bits.

    Which means Comcast would collect on most bits coming in, a major change in how the Internet has always worked, and not just a single dispute over “who peers.”

    Is this a good thing or a bad thing?

    With strong competition, this might be fine because some of the benefit would be passed to the consumer. With a telco/cableco collaborating duopoly, most would go to the shareholders of the companies.

    Given that margins on cable modem service are already 70-90%, I wouldn’t think it necessary to add charges ultimately paid by every consumer.

    1. How can Internet traffic be anything but symmetric overall?

      Every download is somebody’s upload, so in the overall scheme of things, transmitted packets has to equal received packets, less drops.

      Traffic at the borders between ISPs and transit providers is asymmetric, but that’s nothing new, even if the asymmetry is growing because of Netflix et. al. What’s new is that ISPs aren’t just eyeball networks any more, they’re in the transit and CDN games as well.

      So peering isn’t what it used to be, and that’s a good thing.

  10. Richard Bennett Wednesday, December 1, 2010

    I’m glad to see you’ve come around to my point of view – http://gigaom.com/2009/11/22/how-video-is-changing-the-internet/ – Daniel.

    That wasn’t too hard, was it?

    1. I’ve never had your uninformed point of view, Richard. You are a paid lobbyist who has never worked in interconnection. I’ve always been anti-regulation, but I’m not for sale. This is a point of principle.

      1. Now, now, Daniel, let’s not get nasty. I’m not a “paid lobbyist” and never have been, I’m one of the guys who invented Ethernet over twisted pair, the technology that’s employed at Internet interconnects. I gather from your bio that you’re an investor and former market researcher who hasn’t been involved in day-to-day operation of the Internet for what, over a decade? I know you once worked with Vijay at AOL. But we’re not about the bios here, we’re about the quality of the ideas and the validity of the claims.

        You’ve come around to the viewpoint I expressed a year ago on GigaOm to the effect that net neutrality claims were going to appear around peering disputes. This is happening for two reasons: net neutrality is an unusually broad concept, and peering is changing.

        I was disappointed that your article says: “there’s nothing to see here, just a standard peering dispute, move along.” It’s really not a standard peering dispute because it doesn’t involve two backbone transit providers, it involves a CDN and an ISP, so it’s breaking new ground.

        As the Internet adapts to video streaming, the old model where ISPs connect to neutral transit providers who also connect to content providers at a limited number of exchanges is changing. In the new world that’s emerging, the edge is thickening and there is more direct connection and more peering.

        Many of the people who work in interconnection today argue that the traditional “balanced traffic ratio” model doesn’t work any more, since the ratio between CDN and ISP is never going to be anywhere close to balanced.

        It’s too bad you didn’t try to address these current trends in interconnect and their policy implications. Better luck next time, and good luck with your investments.

    2. Richard: very amusing to see you trying to teach grandma to suck eggs. “Thickening of the edge” is not new; I suggest you look up the NANOG27 (February 2003) presentation on Donut Peering by … oh my! Dan Golding!


      Worth noting that at the point of this presentation, it had been well established practice and a major reason for the rise of neutral exchanges … like Equinix – I think you know some folks who were involved early there?

      1. Richard Bennett JZP Sunday, December 5, 2010

        Yes, I know that back in the day when Golding was in the peering business he was certainly aware of the then-current trends. This particular article doesn’t mention them, and that was my point. It’s an odd omission by someone so knowledgeable, don’t you think?

Comments have been disabled for this post