148 Comments

Summary:

The rise of video streaming is dramatically affecting the Internet, according to a two-year study of Internet traffic trends that Arbor Networks recently presented to the North American Network Operators Group. Two years ago, Internet traffic was distributed evenly among a dozen Tier-1 network providers, but […]

cropped_RBThe rise of video streaming is dramatically affecting the Internet, according to a two-year study of Internet traffic trends that Arbor Networks recently presented to the North American Network Operators Group. Two years ago, Internet traffic was distributed evenly among a dozen Tier-1 network providers, but today the majority of traffic flows through direct peering agreements among large content providers, content delivery networks and ISPs. Consequently, Tier-1 networks have shifted their business models from simple packet delivery to richer cloud computing and content hosting services, and new players Google and Comcast have joined the top 10 list of Internet traffic producers — and the more traffic they put on the Internet, the more control it gives them over your online experience.

Traffic is growing much faster than the 50 percent year-to-year rate found by studies such as the Minnesota Internet Traffic Study; yet the “exaflood” of video traffic hasn’t drowned the Internet because network operators have found more efficient paths. The dramatic shift in traffic patterns has to do with the rise of what Arbor calls “the Hyper Giants,” 30 large companies that contribute 30 percent of Internet traffic. Thanks to YouTube, Google alone is responsible for 7 percent of all the traffic on today’s Internet, which puts it in the privileged position of prioritizing its VoIP and video calling services over YouTube without FCC permission.

The onslaught of video is also changing the nature of peering agreements. Traditionally, peering and so-called transit were very distinct from a revenue perspective: Peering agreements were “settlement free” arrangements in which packets changed hands between networks of roughly equal size and scope, but money didn’t. Fee-based network interconnects were confined to “transit agreements” in which a large network operator connected a small player to the entire Internet for a fee; peering is also strictly a “one network to one other network” arrangement. The new wrinkle is “paid peering” agreements in which a large operator permits direct connection for a small fee. Paid peering replaces transit fees that run $2-9 per Mbps with direct connection at $1-3, and enhances service, according to an article on Bill Norton’s “Ask Dr. Peering” web site which explains the value of Comcast’s paid peering and its potential collision with net neutrality regulations:

Paid peering provides better performance than transit, since the traffic takes a less circuitous route. Paid peering allows Google competitors to more easily compete with Google on performance and price without having to reach Google scale.

But paid peering may be forbidden by Question 106 of the FCC’s proposed Open Internet rules because it’s essentially two-tiered network access, Norton points out.

Paid peering illustrates how hard it is to write an anti-discrimination rule for the Internet that doesn’t have harmful side effects for all but the largest content networks. Paid peering is a better level of access to an ISP’s customers for a fee, but the fee is less than the price of generic access to the ISP via a transit network. The practice of paid peering also reduces the load on the Internet core, so what’s not to like? Paid peering agreements should be offered for sale on a non-discriminatory basis, but they certainly shouldn’t be banned.

Video is rising on the Internet, with more of it coming from legal sources such as content delivery networks and less from piracy-oriented systems like eDonkey and BitTorrent. Regulators need to look before they leap into wholesale bans on practices like paid peering that enable the Internet to carry increasing volumes of traffic. The FCC’s last net neutrality order (issued against Comcast in 2008) was an unintentional gift to purveyors of pirated content because it banned P2P throttling; going forward, the FCC should be at least as kind to network operators coping with the rise of video traffic by creative means.

Richard Bennett is a research fellow with the ITIF with 30 years of network architecture experience.

  1. Great read. I am concerned with one thing you make it sounds like everyone uses BittTorrent as this negative thing. Although many people us it to download illegal things there are legal purposes. I know a lot of people that use it for 100% legal purposes. Other then that youtube using 7% of all internet traffic that is insane.

    Share
    1. Most P2P use today is for piracy, but there is a large and significant segment of P2P use that’s perfectly legal. Going forward, when P2P is fully domesticated and doesn’t clog ISP networks as aggressively as it does today, I expect that it will find a niche as the preferred delivery system for ad hoc events like video streaming live news events. It’s not there yet, but the folks at BitTorrent, Inc. and a number of academics are working on refinements to the congestion-sensing logic of P2P systems to move them in that direction.

      As a general rule, P2P is not a very good system for video distribution on the Internet because its use of long, bi-directional paths means that it can take 20-50 times more Internet resources to deliver a file than a CDN does; just count the number of hops a P2P packet has to traverse to get from source to destination and compare them to the hops a packet traverses from a CDN. P2P can place enormous stress on undersea cables, the scarcest resource on the Interenet, and that’s not a good thing.

      Share
    2. The comments from the author clearly determine that he has absolutely no clue about the subject matter (“on-net vs. paid-peering”).

      Look at the source. Bill Norton has made his career on writing white papers on Peering and Interconnection. Does it surprise us that he’s trying to whip up some controversy surrounding this NRPM?

      And to back up Vijay’s satements, Paid Peering is nothing new, it was a service that was offered as far back as ’98-99 (a service that my company offered, and sold). And for you to say this is a new service that is popping up just shows that you haven’t done the most basic research.

      Furthermore, the newer networks who are offering this as a service can only do so because the only very recently (within the last few years) have built national networks!

      Share
      1. Whether or not you believe paid peering is new, the shift of an enormous amount of Internet traffic from transit to paid peering is new, that’s what the data in the Arbor Networks study shows. To ignore this trend in favor of some hair-spitting over terminology shows a lack of willingness to engage in the substance of the debate.

        It’s also obviously not true to say that Norton is professional writer. He was one of the co-founders of Equinix and for a long time a pioneer in building carrier-neutral colos.

        The combination of these two errors in your comment, smeuse, tells me you’d simply rather not deal with the implications of current traffic trends in the Internet core and their potential collision with proposed FCC regs. If the topic makes you uncomfortable, fine, but there’s no need for the personal attacks.

        Share
      2. Daniel Golding Monday, November 23, 2009

        Steve Meuse is absolutely correct. I’m sure, Richard, you know who Steve is, since you have a 30 year history in this industry. At any rate, paid peering is nothing new, nor is there any indication from Arbor’s data that more traffic is traversing it. Bill spent his career writing white papers, not building anything. His job at Equinix was purely evangelical, not operational. Check it out with someone who knows.

        Share
      3. “Hair-splitting” over terminology has major impacts when you are dealing with the FCC. If you get the base assumptions wrong, you are going to have lasting effects of bad legislation….

        Let me pose you this question. What is the difference between Paid Peering and an ISP selling full transit to a content provider, and that content provider filtering out the prefixes they don’t care about?

        Is the FCC going to ban ISPs from selling transit? That kinda breaks things, don’t you agree?

        Again, get the base assumption wrong, the rest of the argument is invalid.

        Share
      4. FWIW, I’ve know Bill Norton for a long time, and I even contributed to some of his earlier documents. I don’t think his most recent work has been grounded in any sense of reality and only seems to exist to stir up controversy where none exists.

        Share
      5. You guys are repeating yourselves.

        Share
      6. Daniel Golding at 12:30 PM on November 23, 2009 wrote:
        “Bill spent his career writing white papers, not building anything. His job at Equinix was purely evangelical, not operational. Check it out with someone who knows.”

        So I guess organizing NANOG doesn’t count for ‘building anything’ in Daniel’s book?

        Irrespective, anyone who’s successfully built a company well understands the integral importance derived from communicating their solution’s core benefits to prospective early adopting customers.

        With the lack of respect insinuated by the ad hominem attacks on Bill Norton’s track record of creating value for others, I can only surmise that Daniel either:
        – Does not carry such experience himself;
        – Is unable to logically, unemotionally challenge Bill’s articulated positions, or;
        – Finds Bill’s writings, which shed light and understanding on how network operators currently commercially integrate with one another (a necessary first step towards more efficiently matching network resources to consumer-driven demands for bandwidth-intensive applications) to be in conflict with his current employer, contractor or other vested interest.

        For the purpose of maximizing value to the conversation, may I suggest to those unproductively exerting energy making ‘playground’ attacks in this comment section focus their attention on the facts and matter at hand? Thank you in advance.

        Share
  2. First you say paid peering may be banned by the NPRM. Then in the next paragraph you jump to the conclusion that you can’t have a non discrimination rule that doesn’t ban paid peering. It seems to me this is a issue that should be raised in the rulemaking, but that the truth is far from certain at this point. I’d think this could easily fall under network management exceptions, but we shall see.

    And what does paid peering’s merits have to do with a prohibition on throttling certain types of applications which have legal uses? Two separate issues, but lumping them together reeks of an attempt to mud pie Net Neutrality in the face.

    Share
    1. I think it’s possible to have a proper non-discrimination rule that doesn’t ban paid peering, but that’s not what the draft rule in the NPRM actually does. If the focus were on offering the service for sale on a non-discriminatory basis, it would be fine; but saying that “all packets are equal” is a big problem for everyone but Google.

      Share
    2. Daniel Golding Monday, November 23, 2009

      Paid peering is not and will not be banned. Bill, unfortunately, made this up. There is no way to read the proposed rulemaking this way – its simply not in the document, at all.

      Share
      1. Actually, it’s perfectly reasonable to read the NPRM that way, not only from Bill’s analysis but on the basis of discussions I’ve had with people at the FCC. The anti-discrimination rule has been kicked around since 2005 as a reaction to former Bell South CEO Bill Smith’s conjecture that ISPs could offer a service that accelerates traffic for paying service provider customers. A major element of net neutrality advocacy is to prevent ISPs from being in that business. Generally, that would entail the ISP offering a CDN or other short path service, and it’s hard to see how paid peering doesn’t fall into that category.

        Paid peering is simply one example of a (perhaps) unintended consequence of an overly-broad anti-discrimination rule.

        Share
  3. [...] en los momentos adecuados, cuando una palabra determinada es mencionada en los mismos. El vídeo es un formato de contenido que está cambiando la fisionomía de la red, y su principal exponente, YouTube, es una empresa que, de acuerdo con uno de sus principales [...]

    Share
  4. My friend Dick Bennett isn’t a lawyer or a policy analyst, so I don’t fault him for misreading the FCC’s NPRM. It most certainly wouldn’t impact peering, or any other paid transport arrangement. Para 106 is about discriminatory treatment for delivery by retail ISPs.

    What Dick didn’t mention is he currently works for ITIF, and his paid by incumbents like Comcast to shill againt Net Neutrality. GigaOm, you should be ashamed for misleading your readers.

    Share
    1. Jane

      It is good to have fly by readers like yourself. If you have been a long time reader of the blog — which you aren’t — then you would know where I personally stand on this issue. And just because I stand on one side of the issue doesn’t mean that I am deaf or blind to other opinions such as Richard’s opinion. I am not sure anyone is misleading anyone here.

      Share
      1. Justa Notherguy Monday, November 23, 2009

        Fair enough, I suppose. And I do think its unreasonable to infer any editorial intention to ‘fool’ the readership. But even this *non*-‘fly by reader’ would suggest that the provided postscript (‘[...] a research fellow with the ITIF with 30 years of network architecture experience.’) just barely qualifies as full disclosure…unless, of course, we presume that all GigaOm readers are either attentive industry insiders or remarkably well-read amateur geeks.

        I’d bet a good number of civilians have more-than-passing familiarity with neutrality issues. Yet these people are hardly likely to have read Om’s opinions, much less to have a remote idea what in heck ITIF is – something, I might argue that _no_ one outside the group truly knows. Heck, even a WSJ op/ed would offer a link or exposition (eg: ‘communication industry think tank’, ‘independent consultancy’, etc.), to point out inherently intrusive business interests.

        Share
      2. Ever heard of Google search, Justa Notherguy?

        Share
      3. Om,

        Actually, I have been reading this site for about 2 years. I certainly am aware of your and Stacey’s positions on network neutrality.

        What I am asking for is some full disclosure on the part of contributors. ITIF, whom Dick works for, was originally founded by ITI, and was at the time a Cisco-IBM-even eBay supported group. But tech funding dried up, and ITIF went with open hand to the Bell companies and to the cable companies. I happen to know for a fact that after the Comcast-Bit Torrent case blew up, NCTA and Comcast funneled a ton of cash to ITIF so they could hire Dick Bennett and George Ou to be flies in the ointment on the network neutrality issue.

        It would help your readers if they knew that this article was written by an author who is paid by Comcast to attack this policy. It doesn’t matter if Dick had these views since he was born, it just matters for transparency’s sake.

        Share
      4. I agree with Jane on this one, Om, though I usually respect what’s on your site. There is little defense in my opinion about not having better disclosure. We come to your site because we trust and respect you, so please don’t spend down that capital you’ve built up by making many mistakes like this.

        On the plus side, you are allowing an open discussion as far as I can tell, so that at least is consistent with my image of your credibility. Hopefully this type of thing won’t happen anymore. Otherwise, I will begin to worry you’ve sold out too somehow, and that will truly be a sad sad day for consumer internet, as you are one of its strongest allies.

        Share
      5. It’s really quite hysterical that so much of the pro-regulation crowd would rather deal in personal attacks than the actual issues, but it’s nothing new.

        Share
    2. The text from the NPRM says:
      “We understand the term “nondiscriminatory” to mean that a broadband Internet access service provider may not charge a content, application, or service provider for enhanced or prioritized access to the subscribers of the broadband Internet access service provider, as illustrated in the diagram below.”

      Jane, it’s an open ended rule that prohibits ISPs from charging content providers for ENHANCED or PRIORITIZED access. That’s an open ended rule that can and will be interpreted by lawyers filing complaints at the FCC who will argue on the basis of the literal meaning of the rule.

      The concern over the current wording of the NPRM is reasonable in the context of Net Neutrality. For example, Rep. Ed Markey’s latest H.R.3458 would ban ISPs from charging content providers for anything. Here’s the exact text.

      “not impose a charge on any Internet content, service, or application provider to enable any lawful Internet content, application, or service to be offered, provided, or used through the provider’s service, beyond the end user charges associated with providing the service to such provider;”

      That bill would go even further to ban devices like the Amazon Kindle since Amazon has to pay Sprint for Kindle connectivity.

      So Net Neutrality proponents have for a long time been trying to ban ISPs from charging content providers under the argument that such charges would be too much of a burden for smaller content providers to overcome. The problem is that this is a foolish and misguided argument because all it does is force content providers to pay more money for inferior transit access.

      Now it’s perfectly possible that this is not the intent of the current NPRM. But why not clarify that?

      But even if this bill doesn’t specifically ban paid peering, it does ban ISPs from charging content providers for prioritization or enhanced access. But I can think of two cases where this would be a good thing. One, an ISP would offer CDN services to the content provider coming though transit. Two, an ISP could offer content providers the option of boosting a broadband subscriber’s bandwidth beyond what the subscriber paid for. That would be no different than Amazon paying Sprint to provide bandwidth to Kindle users.

      So the bottom line is that the NPRM goes too far to prohibit ISPs from providing these services under the assumption that it is always the bad form of discrimination. The more reasonable thing would be to allow ISPs to offer services under non-discriminatory terms.

      Share
    3. I wasn’t aware that we were friends, Jane, but you can never have too many.

      I’ve been commenting on net neutrality since 2002, and only went to work as a policy analyst in June, so my opinions precede my employment by a considerable margin. If you were familiar with the nature of the non-discrimination rule as it’s been discussed in DC and in Europe since 2005, you’d be aware that it bans payments to an ISP – such as Comcast – by a service provider that will give the service provider better QoS than would be afforded to a service provider with no specific contract with the ISP. Because paid peering is a short path to the consumer, it’s also a high-priority path, due to the statistical nature of the Internet and the impact of Round-Trip Time on TCP. So your analysis is clearly wrong, and you resort to smear tactics to hide its weakness.

      But thanks for playing.

      Share
    4. Any by the way, the tag line to my post discloses my employment, so I don’t know where you get off saying I don’t mention that work for ITIF. I do, and ITIF has many sponsors on all three sides of the net neutrality debate.

      Here’s a clue, no charge: Bring some substance.

      Share
      1. Justa Notherguy Monday, November 23, 2009

        @Richard bennett:

        > [...] ITIF has many sponsors on all three sides of the
        > net neutrality debate.

        Really? And who would those ‘sponsors’ be? Last I checked, ITIF have steadfastly refused to reveal their sources of funding. Nor have I found any such information on-line, via SourceWatch, etc.

        Therefore, I’d very much like to see a list of said sponsors, showing the exact nature of their relationship(s) – money? technical help? good wishes? – with ITIF. And I would bet that I am not alone in this.

        Given your aggressively defensive attitude, here, I anxiously await _your_ ‘substantial’ response.

        Share
      2. I don’t do fund raising or public relations, so I couldn’t give you the full list of who funds ITIF even if I wanted to. But I will reiterate that I haven’t modified my views as a consequence of my employment, as everyone who’s followed my writing over the years can attest.

        So rather than mud-slinging, why not argue a real point?

        Share
      3. Justa Notherguy Monday, November 23, 2009

        Richard Bennett wrote:
        “[...] ITIF has many sponsors on all three sides of the net neutrality debate. ”

        Intrigued, Justa Notherguy posited a logical follow-up:
        “Really? And who would those ’sponsors’ be? Last I checked, ITIF have steadfastly refused to reveal their sources of funding.”

        Richard Bennett answered:
        “I don’t do fund raising or public relations, so I couldn’t give you the full list of who funds ITIF even if I wanted to.”

        To which, I respond:

        I never asked for a/the ‘full list’. A representative one will do, just fine…say, one well-known sponsor from each of the ‘three sides’ you cited.

        Neither did I ask what you do/don’t do, at ITIF. Let’s recall that _you_ volunteered the varied nature of ITIF’s sponsors, as affirmative defense of both their objectivity and your own. Thus, my interest in seeing some examples from that diverse group. Ipso facto.

        So, if you now hope to tap-dance around this subject (‘[...] why not argue a real point?’), here’s a clue – no charge: save the puerile, Debate Club distractions for a less-rigorous venue. You claimed evidence of objectivity; I asked for it. Bring some substance.

        Share
      4. You’re the one who’s dodging the issues, anonymous Justa Notherguy, not me. I’m using my real name as I always have on the Internet, and posting my own opinions. You’re claiming that my opinions are bought and paid for and demanding full disclosure of my current employer’s sponsors, while you too much of a coward to even post blog comments using your real name. That’s a bit of double standard, don’t you think?

        Share
      5. Justa Notherguy Monday, November 23, 2009

        Richard Bennett wrote:

        > You’re claiming that my opinions are bought and paid for [...]

        No. I never wrote any such thing, nor have I implied it. But anyone can see that, just by reading my responses (above). Come on, now – this saber-rattling is merely yet another distraction from my question. You implied the diversity of ITIF’s sponsors-list is evidence of objectivity. Ok, then…show me. That’s all I’ve asked.

        > [...] and demanding full disclosure of my current employer’s
        > sponsors [...]

        Not ‘full’ – just a few. Three, in fact. Have you neglected, to re-read my previous response? Besides, it was you who brought them up in the first place. Remember? Up to that point, it would never have occurred to me to ask. Must be a bit embarrassing, knowing that you put yourself in this position, to no real purpose.

        > [...] while you too much of a coward to even post
        > blog comments using your real name.

        Tsk, tsk…name-calling? Richard, really! I expected more from a man of your background and education.

        > That’s a bit of double standard, don’t you think?

        This from a guy who volunteered un-named parties as references for his veracity, only to admit (see prev.) that he has no such names to provide and couldn’t get them, even if he tried? Then, when asked to explain this discrepancy, rather than admit to his error(s) he starts flinging epithets? And you see a ‘double standard’…where? LOL

        Seriously, Richard, you need to get a grip. I believe this kind of behavior reflects poorly on your employer, as well as on yourself. And, heaven knows, ITIF seems to have enough detractors. Surely, they don’t need the additional burden of being associated with schoolyard grandstanding and shameless weaseling.

        Share
      6. @Anonymous Justa Notherguy:

        If you’re not implying that my opinions are bought and paid for, why do you keep harping on ITIF’s sponsors? You know as well as I do that ITIF’s policy is not to disclose, so I couldn’t tell you who they are if I wanted to.

        Given that you refuse to give your name or say who pays you, I’d think you’d be sympathetic to ITIF’s policy in this respect. Well, you would be except for some sort of aversion to consistency.

        So come on now, I’ve disclosed by name and my employment, and you can visit my web site, bennett.com, and read my resume and my past writing on this and other subjects.

        Share
      7. I wrote the previous comment, for some reason the blog didn’t pick up my Facebook login.

        Share
  5. [...] guest blog at GigaOm deals with paid peering and the net neutrality regulations, How Video Is Changing the Internet: But paid peering may be forbidden by Question 106 of the FCC’s proposed Open Internet rules [...]

    Share
  6. [...] Bennett also wrote a good explanation here at GigaOm on the changing landscape of the Internet and why it is a bad idea for the FCC to prohibit paid [...]

    Share
  7. [...] How Video Is Changing the Internet [GigaOM] [...]

    Share
  8. A few clarifications:
    “Thanks to YouTube, Google alone is responsible for 7 percent of all the traffic on today’s Internet, which puts it in the privileged position of prioritizing its VoIP and video calling services over YouTube without FCC permission.”
    I wasn’t aware that since I’ve been prioritizing vonage over Hulu on my home network using my linksys WRT, I was in violation of the law. Perhaps richard can help me get the proper permission from the FCC? In the meantime I’ve reverted my home network QoS settings back to factory default.

    “The new wrinkle is “paid peering” agreements in which a large operator permits direct connection for a small fee. ”
    Perhaps richard isn’t aware, but ISPs have been providing “on-net” access as a form of limited transit product for over a decade, I know I had turned this on for customers as far back as 1999.

    In the meantime, this line demonstrates quite clearly the competence and level of research in both articles:
    “Google wants Paid Peering to be illegal, along with any form of access tiering. ” from the dr. peering article.
    I haven’t seen a statement to that effect anywhere yet, perhaps Richard can point that out?

    /vijay

    Share
    1. Excellent snark from Google’s Director of Net Ops.

      1. The home network point is one that I made to the FCC as a witness at the hearing on P2P at Harvard, because it illustrates the gap between current practice and the rules Google’s Rick Whitt and others have petitioned the FCC to adopt. It’s nice to have my arguments thrown back at me, but you and Rick should talk.

      2. The paid peering service is new for the major ISPs, but not unprecedented in world history.

      3. Google’s opposition to enhanced interconnect for a fee is a matter of clear public record. See Rick Whitt’s statements of the Google Policy blog, http://googlepublicpolicy.blogspot.com/2007/06/what-do-we-mean-by-net-neutrality.html :

      “…in Google’s view, what should the broadband carriers not be allowed to do? …

      * Levying surcharges on content providers that are not their retail customers;
      * Prioritizing data packet delivery based on the ownership or affiliation (the who) of the content, or the source or destination (the what) of the content; or
      * Building a new “fast lane” online that consigns Internet content and applications to a relatively slow, bandwidth-starved portion of the broadband connection. ”

      As you’re a technologist, Vijay, these prohibitions don’t make sense to you, but they are the program your employer is seeking to impose on their competitors. Like the unbindling issue we’ve discussed before, they compromise the future to protect the status quo and are therefor bad for progress.

      Share
      1. Actually I’d be a lot more interested if you would answer the question I asked, not what you want to say: Specifically I asked (quoting you) the following: “Thanks to YouTube, Google alone is responsible for 7 percent of all the traffic on today’s Internet, which puts it in the privileged position of prioritizing its VoIP and video calling services over YouTube without FCC permission.”
        How does that actually work again? Alternatively, how does one get from a claimed 7% of traffic to the statement “privileged position of prioritizing its voip and video calling over youtube without fcc permission?” And could you help me with how I need to comply with the FCC since I didn’t ask their permission for my home network.

        As for “2. The paid peering service is new for the major ISPs, but not unprecedented in world history.”
        Again, do you actually know what you talking about? I said I was turning up paid peering in 1999. On-net or limited transit was being sold by all major ISPs back then.

        As for #3, again, I don’t see anything about paid peering.

        Share
      2. Maybe my sentence wasn’t clear. As a private network operator, Google doesn’t need FCC permission to prioritize (by order) the traffic that leaves its network in any way that it sees fit; it’s exempt from NN regs since it’s not an ISP.

        If Google wants to offer VoIP in the future (in conjunction with Google Voice, for example,) it can ensure that its own VoIP packets are interleaved with YouTube in such a way that YouTube doesn’t cause excessive jitter. This makes perfect sense, and the FCC won’t complain. But Google’s proposed regulations could make it difficult for Comcast, say, to provide Vonage with the same service. A bit inconsistent, don’t you think?

        How old is Paid Peering? Bill Norton is generally regarded as an authority since he co-founded Equinix and has more experience in the area than anyone else, so this will be he said-she said for the time being. If you have some data, please share.

        The words “paid peering”aren’t in Rick’s statement, but the effects he lists are consequences of it. In the policy world, we tend to think abstractly.

        Share
      3. “How old is Paid Peering? Bill Norton is generally regarded as an authority since he co-founded Equinix and has more experience in the area than anyone else, so this will be he said-she said for the time being. If you have some data, please share.”

        Co-founding equinix and actually taking two ISPs to “tier-1″ status in us and eu isn’t exactly the same thing. Like I said, I’ve been turning up paid peering or on-net routes as a product since 1999 and so have other ISPs. Perhaps you should talk to someone who actually does this for a day job vs writing white papers?

        Lets try this one more time shall we? “how does one get from a claimed 7% of traffic to the statement “privileged position of prioritizing its voip and video calling over youtube without fcc permission?””

        “The words “paid peering”aren’t in Rick’s statement, but the effects he lists are consequences of it. In the policy world, we tend to think abstractly.”
        Would that we actually thought at all. Like I said, perhaps you could boil it down to concrete words that idiots like me can understand, not being up with high faultin “policy world” thinking.

        /vijay

        Share
      4. I’m not sure that your “on-net routes” is the same product as the Paid Peering that Nortion is interpreting; the Arbor study found a large increase in the traffic that moves through these transit bypass paths, and that’s the actual story. While this service may have been available for a while, its use is radically increasing. That’s data, BTW, not anecdote, so if you have a problem with the Arbor data, you’ll need some data of your own to refute it.

        The significance of the 7% isn’t that difficult, is it? If you pump the ISP’s pipe full of packets of all kinds, you’re going to cause jitter for third party services whenever you deliver a clump of packets back-to-back. You can order your own voice packets optimally in each clump, as you’re immune from FCC regulation, but Vonage has to deal with the jitter you cause. How are they going to do that without a little help fro the ISP?

        Paying for a prioritization service – even as Paid Peering – would probably be forbidden under the regulations Rick wants. Do you read the statement seeking a ban on “Prioritizing data packet delivery based on the ownership or affiliation (the who) of the content, or the source or destination (the what) of the content” differently? If so, what does it say to you?

        Share
      5. “I’m not sure that your “on-net routes” is the same product as the Paid Peering that Nortion is interpreting; the Arbor study found a large increase in the traffic that moves through these transit bypass paths, and that’s the actual story. While this service may have been available for a while, its use is radically increasing. That’s data, BTW, not anecdote, so if you have a problem with the Arbor data, you’ll need some data of your own to refute it.”

        I think that is the entire problem statement right there – “I’m not sure that your “on-net routes” is the same product as the Paid Peering that Norton is interpreting”
        I think you actually don’t know what you are talking about basically, so there isn’t much further point in discussion, but for completenesses sake, I’ll go through your points.

        “The significance of the 7% isn’t that difficult, is it? If you pump the ISP’s pipe full of packets of all kinds, you’re going to cause jitter for third party services whenever you deliver a clump of packets back-to-back. You can order your own voice packets optimally in each clump, as you’re immune from FCC regulation, but Vonage has to deal with the jitter you cause. How are they going to do that without a little help fro the ISP?”
        This one is relatively easy – once the traffic hits the ISP, it is all the same to them, voip, video etc. When they get congested they are doing to drop packets at random when the queue gets full. The standard drop discipline is juniper routers which are in every ISP bar one or two, is RED, which stands for RANDOM Early Drop. It would be sort of hard to predict ahead of time and re-order packets such that a *random* drop could be predicted ahead of time. If I could do that, I’d be in the futures business, not moving packets around.

        “Paying for a prioritization service – even as Paid Peering – would probably be forbidden under the regulations Rick wants. Do you read the statement seeking a ban on “Prioritizing data packet delivery based on the ownership or affiliation (the who) of the content, or the source or destination (the what) of the content” differently? If so, what does it say to you?”

        So now we are in to the “probably” now. I did read the statement on prioritizing data packet and affiliation – nowhere does it say to me you can’t conduct a commercial transaction for getting access to a providers routes by direct interconnection. Thats a commercial transaction.

        Share
      6. Daniel Golding Monday, November 23, 2009

        With all respect to Bill, he is not a co-founder of Equinix. The founders of Equinix were Jay Adelson and Al Avery.

        And, he is wrong about paid peering, wrong about Google’s intent, and wrong about the proposed rulemaking. I have told him this directly.

        I am not an employee of Google and I take no money from them. Heck, I don’t even agree with them on Net Neutrality. But I do think that this sort of misinformation is terribly harmful to the debate. I suggest asking people other than “Dr. Peering”.

        Share
  9. Once again,complete wrongness. Bill Norton is the ONLY person in the Internet industry who thinks 106 affects peering. One source is extremely poor reporting, but as this is apparently an editorial sloppy/no research is OK [sorry, I too am a 'fly-by' reader because I'm issue focused; google alerts tell me abut relevant stuff interesting to my niche, so not I'm not a faithful lapdog^Wreader anywhere, nor do i wish to be...].

    From the NPRM:
    “We understand the term “nondiscriminatory” to mean that a broadband Internet access service provider may not charge a content, application, or service provider for enhanced or prioritized access to the subscribers of the broadband Internet access service provider, as illustrated in the diagram below.”
    Paid peering, which is not new and has nothing to do with video, does not prioritize to *subscribers*, therefore there is no applicability. If Richard thinks it is new to “major ISPs” then maybe he should say whichones. Likely they are minor or telephant dinosaurs who were behind the curve for the last decade. Direct interconnection to a network is much further away from the subscriber edge. Anyone with network design clue would get this.

    As to this lovely bit: “not impose a charge on any Internet content, service, or application provider to enable any lawful Internet content, application, or service to be offered, provided, or used through the provider’s service, beyond the end user charges associated with providing the service to such provider;” This text is more likely to cause issues, but if is is a different service offering, and is optional, is that IMPOSED? I think not, but IANAL.

    “Because paid peering is a short path to the consumer, it’s also a high-priority path, due to the statistical nature of the Internet and the impact of Round-Trip Time on TCP. So your analysis is clearly wrong, and you resort to smear tactics to hide its weakness.”
    Short != high priority. If you honestly buy this argument, then you must desire legislation of queing disciplines and specific TCP implementations. Completely ludicrous.

    I think this article is wildly inaccurate an providing distribution for a broken interpretation of the proposed rule making.

    Share
    1. Obviously, Bill is not “The Only Person in the Internet Industry” who thinks 106 might have an affect on peering and transit. Try to make a serious argument next time.

      Share
      1. Name someone else then, Anonymous. I’ll happily ask them to actually read 106 and look at the diagram that clearly states “End user’s broadband Internet access service provider may not charge CAS provider for enhanced or prioritized delivery to end user over this link.” where “this link” is the last mile, where all QoS discussions have been based.

        Share
      2. All that’s necessary to for the FCC to apply the non-discrimination rule to the peering side of an ISP network is the realization that the Last Mile and the Middle Mile are seamless parts of a single network. You don’t have to be very smart to make that leap.

        Share
      3. That would then be a different entity than THIS proposed rule-making, which clearly points to the last mile. Should there be a different proposed rule-making, the concern might have a shred of weight, but that would extend beyond where the scare-resource, duopoly claims have a reasonable foothold … the justification forr this whole pile of junk.

        Share
  10. Wouldn’t it be nice if GigaOm had someone on staff (or even in consultancy) who actually knew how the internet worked? And I don’t mean like “click the link, look at the site”, I mean more like “hey, the internet has packets, how do they get to where they go”.

    Seriously guys, please get some clues.
    Articles like this are why network engineers think you guys lack credibility.

    Share
  11. I find it interesting that P2P is really decreasing as viewers want to stream content, and not always download content to their computers. With new viruses coming out every day I certainly can’t blame users for doing this and I assume it’s a trend that will continue.

    Share
  12. Patrick W. Gilmore Monday, November 23, 2009

    I like Bill, he’s a nice guy, and a friend of mine. I’ve helped him with many of his white-papers and he has thanked me publicly, so I am not unfamiliar with his work. However, he is, unfortunately, occasionally confused about certain things.

    I do not know you, but you claim to have 30 years of experience and put yourself out as an expert in the field. Then you make statements which prove you are quite ignorant of the topics at hand (e.g. not knowing “on-net routes” and “paid peering” are the same as “6” and “half dozen” to anyone with more than a passing familiarity with the subject matter).

    My _guess_ is that you suffer from the same problem as Bill. You have talked to a lot of people, you have hung out at conferences, you have read about the topic, etc., etc. But you have never logged into a single router, never run a network, never had to make decisions while live traffic was running over the equipment being modified, and in general never had to actually deal with these issues “in real life”. I could be wrong, but it would be surprising that you truly are experienced yet make such sophomoric errors.

    People like JZP, Daniel Golding, Vijay Gill, and I are experienced with these subjects. While we are all experienced, our histories and points of view are about as widely separated as it is possible to be. Yet we are united in believing that you and Bill are both wrong on several objectively provable points. Perhaps faced with a rather large contingent of experience and viewpoints, you should re-consider your stance? Or at the very least stop trying to win semantic arguments when we are trying to educate you.

    For instance, Vijay has actually done what and when you claim was not done. Is it not better a better strategy to listen and learn, rather than try to argue over the meanings of individual terms? Especially when you are wrong on those meanings?

    Your blog, your choice. However, I find people get farther working together. Would you consider working with us?


    TTFN,
    patrick

    Share
    1. Interesting comment, Patrick, and one that illustrates the problem with having this kind of discussion. I’m not a network operations person, for sure, and I don’t claim to be. I’m one of the people who first created Ethernet over twisted pair, Wi-Fi, and some related LAN and WLAN things. I’ve been around network protocols and standards for a long time, but not in an operations role. So my interest in the subject comes from observing how the Internet is adapting to changed patterns of usage, and how the FCC’s proposed rules may affect it. These are not questions that one needs to be a network admin to consider, are they? And in fact being too close to the action – as Vijay clearly is – can warp one’s perspective to the point where it’s difficult for him to consider the wider implications either.

      It doesn’t really matter, for purposes of this discussion, whether some ISPs started selling paid peering in the last couple of years or ten years ago as much as it matters that service providers started buying and using it on a large scale. Vijay wants to obsess over the history of the service, I want to talk about the adoption of it. So to jump up and down and complain as he is is equivalent to attacking spelling or grammar; it’s a nit-picking detail that doesn’t go to the actual issue. And you can understand why: it’s embarrassing to Google to find that there really is (or may be, if you will) a self-serving element to the Non-Evil company’s long-standing support for a highly-regulated Internet.

      And similarly, the ability to interpret the text of the proposed rules in the NPRM is not something that comes from network operations experience, it comes from an understanding of how the FCC works, the history of how similar rules have applied in the past, and the nature of the commercial transactions and packet flows that they affect.

      Knowing the Cisco CLI doesn’t really arm a person to deal with these questions, so let’s not pretend otherwise.

      Share
      1. > Knowing the Cisco CLI doesn’t really arm a person to deal with these questions,
        > so let’s not pretend otherwise.

        And having protocol history has nothing to do with operations and policy. Those of us who have operated networks in the US have dealt with the impact of policies (pre & post telecom act of 96, pre & post data CALEA, pre & post interLATA releif…) and are not blinded by such experience. Rather, we are informed by our legal teams, years of intercompany co-operation, years or talking with LEOs and regulators, time spent working with data collection, on federal advisory councils, etc. We *are* the people who know what we’re doing, and we call BS.

        Please do address any of my previous comments regarding specific language. The language is targeted for “access to the subscribers of the broadband Internet access service provider” which has diddly to do with paid peering.

        Share
      2. Richard Steenbergen Monday, November 23, 2009

        This has nothing to do with knowing how to configure a router, this has to do with having actual experience with peering and operating a network from a policy perspective. Now don’t get me wrong I like Bill Norton personally, I think he’s a really nice guy and I wish him the best, but at the same time he has been in way over his head trying to write about an industry he doesn’t really understand for many many years. From a technical perspective he has absolutely no clue what he is talking about, he is simply regurgitating what others have said with no ability to distinguish fact from fiction.

        You would do well to listen to the insightful comments posted here, by people who have doing more with peering than you will ever dream of understanding. Vijay, Dan, and Patrick are not only some of the most authoritative sources on peering anywhere in the world, but they also happen to be correct this time too. :)

        Share
      3. …I’m not a network operations person, for sure, and I don’t claim to be. I’m one of the people who first created Ethernet over twisted pair, Wi-Fi, and some related LAN and WLAN things. I’ve been around network protocols and standards for a long time, but not in an operations role.
        [...]

        well, in your resume (http://bennett.com/resume.pdf) you state you are a “broadband network architect” and a “network engineer”

        how can one fulfill either of those roles, if they do not understand basic interconnection methods such as paid-peering/partial/on-net routes

        you should have some understanding about the operation of a broadband network and how it interconnects with other networks, if you are a broadband netarch..

        Share
      4. Patrick W. Gilmore Monday, November 23, 2009

        You are very good at ‘nit picking’ while missing the wider point, which is pretty ironic given your reply.

        Do you honestly believe Google’s Director of Network Operations spends time logging into routers these days? Of course not. Neither does Mr. Golding or I. (I do not know if JZP does these days.) But without some experience in how networks work, and whether something is useful in a real, live, business setting, it is impossible to make educated decisions regarding those things.

        I have completely and perfect faith in Mr. Golding’s ability to read the proposed rule and tell me whether it means what you say it means. I have complete and perfect faith in Mr. Gill’s ability to tell me whether something happened and whether it was common. I have complete and perfect faith in JZP’s judgement in whether a shorter path is “high priority” in an _operational_ setting.

        More importantly, my own opinion concurs with theirs. Additional commentary by people like Steve Meuse only emphasize the point.

        Alternatively, you make claims based on papers which are written by people who have exactly zero experience with the subject matter. You argue that X is important or new when those who actually have to run a real network with real traffic say otherwise. You buttress your claim with _factual_ errors, on points which are not open to interpretation. When we point out these errors, you ignore them or further populate your blog with additional errors trying to cover your original error.

        I am confused why you think this is a useful course of action?

        Let’s be clear about just a few things:

        1) Wide adoption of paid peering is not terribly new.
        2) The “Tier Ones” have not been the center of the Internet for years.

        It doesn’t matter how smart you are, if you base your logic on faulty assumptions, your conclusion is almost certain to be wrong. Anyone with any real operational experience would know these points without having to read a paper by Mr. Norton. That you read the paper and claim otherwise only makes anyone who has actually done this work think poorly of you.

        Still think that operational background is still irrelevant to the question at hand?

        That said, it is true that operational experience is not required to read a proposed rule. We disagree on what the rule says. But even if you can read the rule, I have zero faith you understand the repercussions of the rule. Not only do you have no experience in the subject, but you have proven you cannot take instruction from those who do. So while you may be able to read the rule, why does that matter if you cannot explain what the rule will mean to those actually doing business on the Internet?

        In short, you add no value. Just the opposite, you confuse the issue, which is bad for everyone.


        TTFN,
        patrick

        Share
      5. Actually, Patrick, I have another source on paid peering, a paper presented at TPRC 2007 by several people from MIT and one other author. See: http://people.csail.mit.edu/wlehr/Lehr-Papers_files/Clark%20Lehr%20Faratin%20Complexity%20Interconnection%20TPRC%202007.pdf

        This paper says: “This line of thought has two implications—more formal peering contracts and the emergence, as we will discuss below, of paid peering” and “The two new types of agreements we will discuss are paid peering and partial transit” and “The emergence of paid peering, although non-standardized in its details, is not surprising as a concept”.

        This suggests that paid peering is “a new wrinkle” in the Internet interconnection space. Oh, BTW, the other author of that paper is somebody named Patrick Gilmore; that would be you.

        Does any of this ring a bell?

        And BTW, I don’t buy the argument that “paid peering” is the same thing as “on-net access”. Paid peering is going to take you into one and only one network, while on-net access simply gives you a closer point of presence to connect with a transit network, which is not the same thing at all.

        So no, I don’t know that on-net access is the same thing as paid peering; in fact, I know that it isn’t.

        Share
      6. Richard Steenbergen Wednesday, December 9, 2009

        Richard,

        You don’t “buy” the argument that paid peering is the same as on-net? Are you out of your #$%^&ing mind? Do you think there is some kind of conspiracy theory to hide the name from you? Let me clear this up for you with some definitions:

        Peering is when two networks agree to exchange on-net (i.e. their customer) routes with each other for free.

        On-net transit is when one person pays money for on-net (i.e. their customer) routes.

        What do you think paid peering is? That’s right, money in exchange for on-net routes. You fail at even a basic understanding of the concepts, let alone the terms. Please, I’m begging you, stop embarrassing yourself.

        Share
      7. Sorry, Steenbergen, but you’re wrong again. A range of “On-net services” that can be purchased from large network operators, some of which are simply interconnection and routing within the large operators own space, some of which are transit to the whole Internet, and some of which are combinations of carriage and routing. Level 3 has something they call “Extended on-net access” which is both carriage and transit: http://www.level3.com/index.cfm?pageID=454

        You insist that these terms have the meanings you want them to have, but the industry has adopted a set of meanings that overlap with yours only slightly. Given the many competing authorities in this field, and the wide range of uses among providers, I’ll stick with the definitions I’ve found from credible sources.

        Share
      8. Richard Steenbergen Wednesday, December 9, 2009

        Oooooook wait, I think I see the problem. You don’t understand the difference between on-net in a transport sense (i.e. a building you service to via your own network, without having to buy a tail/loop from another carrier) and on-net in an IP sense (your customer routes). One of these things is not like the others, one of these things just doesn’t belong. Again, basic concepts 101.

        Share
      9. In other words, “on-net” as a prefix implies many different things, some of them very much like peering, some like transit, and some like carriage. Because of that ambiguity, it’s best to use the term “paid peering” when we’re talking about a service in which a content provider pays an eyeball network for access to the eyeball network’s retail customers.

        Most credible sources – from David Clark to Patrick Gilmore to Bill Norton – regard paid peering as an emerging and valuable phenomenon.

        Share
      10. Richard Steenbergen Wednesday, December 9, 2009

        In other words, there are some concepts which share similar words from the english language, but which are entirely different things with absolutely no relationship to each other. Those of us who understand the meaning of the words we are trying to use know the difference, while those of us that don’t (that would be you) flail wildly in a sea of fail.

        On-net as a prefix implies that something is… yes you guessed it, ON YOUR NETWORK. When you’re talking about a building that is on your network, it’s called an on-net building. When you’re talking about an IP customer route which is on your network, it’s called an on-net route. But BUILDINGS ARE NOT THE SAME THING AS THE INTERNET ARE THEY?

        I’m going to stop talking to you now, because this is absolutely pointless. It’s like arguing with a 4 year old, it doesn’t matter if I’m right because you wouldn’t know it if it bit you in the ass. You have absolutely no knowledge basis to be making any claims or conclusions about peering or interconnection on the Internet in general, but at least you’ve done a great job embarrassing yourself and proving that point. :)

        Share
      11. That’s a good move, Steenbergen. Please note that I didn’t use the term “on-net” on my blog post, I used the precise terms such as peering, paid peering, and transit. “On-net access” and “on-net routes” were introduced in the comments by Vijay Gill in some sort of attempt to prove that my claim that paid peering is a new wrinkle is false. Gill’s claim, even if it were true, would be beside the point. The FCC regulations seek to broadly ban arrangements under which content or service providers have enhanced access to eyeball network customers. It’s plausible to consider paid peering as one such arrangement, and most responsible parties agree that this would be a bad outcome.

        That’s the point of my blog, so this terminology war is simply obfuscation.

        Share
  13. [...] Life isn’t just getting harder for device manufacturers, it may soon become difficult for smaller content distributors to negotiate cheaper and more effective bandwidth over the Internet as Google and other Net Neutrality proponents are pushing to ban Internet Service Providers (ISP) from charging content providers for enhanced or prioritized access.  While it might sound like a noble effort to protect content providers from having to pay ISPs for access to broadband subscribers, the effect is that content providers lose access to a cheaper and more reliable form of bandwidth.  If the new FCC NPRM regulations are passed in its current form, only companies on the scale of Google will be able to build effective content distribution networks because they are large enough to negotiate free peering. [...]

    Share
  14. JZP demands comment on: “access to the subscribers of the broadband Internet access service provider” which (in his opinion) has diddly to do with paid peering.

    You really don’t see this? It’s a mystery ow an ISP who sells a service (whatever you want to call it) that provides faster and higher-capacity access to the ISP’s network is not providing faster and higher-capacity access to the ISP’s subscribers?

    WTF, dude.

    Share
    1. Well, “Dude”, your argument means that any broadband ISP can sell no other internet access. Do you honestly believe that limiting the business opportunities of broadband ISPs large and small is the intent? You think that Version Business will be interested in shedding the as701 enterprise customer so that they are not providing “preference” to the path between people buying their service?

      The language is specific regarding broadband subscribers and QoS (enhanced or prioritized access). The reference diagram is clearly the last mile. FFS it says “End user’s broadband Internet access service provider may not charge CAS provider for enhanced or prioritized delivery to end user over this link”. [diagram 2, page 42, where you can find the text which has been quoted several times to you.

      That is plain as day. Not even a regulator could contort that to the meaning you and Bill seem to wish there to be.

      Share
      1. See my previous reply to your first comment implying that ISP Last Mile and ISP Middle Mile are separate networks, JZP.

        The end user’s broadband experience is affected by the nature of the interconnect between the ISP’s network and the service provider’s network; Vijay’s employer didn’t spend $10-15 billion on an unregulated private network and its peering and transit agreements because it didn’t make any difference.

        Look, I think it’s really noble that Vijay has come in here guns blazing to defend the honor of his employer, and that his friends are standing up for him, but running the discussion down a rat hole of terminology isn’t really advancing the discussion or helping to move the US toward a good Internet policy framework.

        The issue underlying this discussion is how much freedom the ISPs need to have in terms of interconnect agreements to provide the needed Quality of Service to diversify Internet applications in the future. The network neutrality advocates (and some in the operations community) argue that the best way forward for the Internet is to keep on doing things the way they’ve always been done; Vint Cerf argues, for example, that treating all packets in the same, “Best Efforts” way has given rise to a rich set of applications, and therefore has great utility. On the other side, people who want to see more diverse applications on the Internet (those that require QoS or volume-sensitive discount pricing) argue that allowing ISPs to sell interconnect at varying levels of QoS would be beneficial.

        The Paid Peering example illustrates that business arrangements that provide poor man’s QoS by offering a short path (TCP throughput is inversely proportional to RTT, all other things being equal) show that such arrangements are beneficial and should be encouraged. Some net ops people are simply phobic about change, since they’re so busy bailing water under the current scenario. I understand that.

        But Net Ops people aren’t the best people to dictate policy to the FCC. As one of you has pointed out, you learn about regulations from talking to your legal staffs, and the legal staffs learn what to teach you from seeing how the FCC and the courts interpret policy. This system is inherently retrospective. The questions before the US are prospective, namely which network operations and business policies stimulate innovation and wide adoption of broadband and which don’t. There’s a great body of empirical and analytical literature that shows that the policies that Vijay’s employer advocates – open access, un-bundling, line sharing and single service level – inhibit broadband adoption in both the current (copper-based) generation and in the fiber-based next generation.

        These are the issues that the FCC’s proposed rules will impact, not whether the right term for direct access to an ISP network is “on-net routes” or “paid peering.” We’re talking about the direction in which we want to take network policy, not the place that it comes from.

        Share
      2. Once again, the blog ate my name; the previous comment is mine.

        Share
      3. Yeah and once again, reply buttons come and go…

        > See my previous reply to your first comment implying that ISP Last Mile and ISP
        > Middle Mile are separate networks, JZP.

        From a regulatory perspective -both as proposed here and as implemented in many
        countries- that is not relevant as those network elements are traded in an open market.
        You are sidestepping the point that 106 does not say anything on-point to external
        network edges. There *might* be any number of changes proposed, but *might* isn’t
        what is on the table. What is on the table expressly points to the last mile, and you
        keep failing to acknowledge that.

        > The end user’s broadband experience is affected by the nature of the interconnect
        > between the ISP’s network and the service provider’s network;

        You are creating dichotomies where there aren’t any. You avoided touching my VZ
        example; why? Without a strictly segmented and regulated last mile business entity,
        how can you claim that there are such things as broadband providers with no other
        content or connectivity lines of business?

        > Vijay’s employer didn’t spend $10-15 billion on an unregulated private network
        > and its peering and transit agreements because it didn’t make any difference.

        So you have a bone to pick with every unregulated packet switched network that
        exists? Neat, and the answer is to regulate them all?

        > Look, I think it’s really noble that Vijay has come in here guns blazing to defend
        > the honor of his employer, and that his friends are standing up for him, but
        > running the discussion down a rat hole of terminology isn’t really advancing the
        > discussion or helping to move the US toward a good Internet policy framework.

        Colleagues concerned about people propigating misinformation are not friends
        standing up for anything but intellectual honesty. If you bothered to look up the
        people commenting, you will note they are long-standing members of the operations
        and engineering community with a diverse set of opinions. We happen to agree on
        9.8m/s^2(rounded) as standard acceleration due to gravity, but somehow defending
        that fact doesn’t require “nobility”.

        > The Paid Peering example illustrates that business arrangements that provide
        > poor man’s QoS by offering a short path (TCP throughput is inversely proportional
        > to RTT, all other things being equal) show that such arrangements are beneficial
        > and should be encouraged. Some net ops people are simply phobic about change,
        > since they’re so busy bailing water under the current scenario. I understand that.

        Once again, you ignore the question raised by smeuse, myself and others. If you
        buy this argument then how can any transit sales not violate such principles? There
        is no differentiation visible externally between a standard transit agreement and any
        paid peering agreement, so address how you think proposing such a restriction on
        commercial offerings will fly with any business?

        > But Net Ops people aren’t the best people to dictate policy to the FCC. As one
        > of you has pointed out, you learn about regulations from talking to your legal staffs,
        > and the legal staffs learn what to teach you from seeing how the FCC and the courts
        > interpret policy.

        It was me and I said “informed” and was in the context of not being blinkered nor
        blinded. Most of the folks participating on this thread have been the ones *instructing*
        legal staff over the past few decades.

        I personally don’t care what google or comcast advocates. Nor do I see how either is
        relevant to your claims that:
        – google can magically prioritize traffic on other folks’ networks
        – “recent” video is the cause of paid peering
        – broadband companies can’t offer any other connectivity without being unfair
        – …I’ve lost track of what else I need to correct; sorry

        > These are the issues that the FCC’s proposed rules will impact, not whether
        > the right term for direct access to an ISP network is “on-net routes” or “paid
        > peering.”

        The proposed rules in front of us do not deal with the non-access network
        elements; where is the text that says otherwise? Thus far when pressed you
        have said rules “might” be changed, etc. Other than maybes and mights, where
        is the basis for the concern? Smells like floating the bogus idea only to champion
        protecting all of us from it.

        Have a good night!

        Share
      4. Richard Steenbergen Monday, November 23, 2009

        Point out an actual error? You didn’t even know what “on-net” and “paid peering” were, that alone should be all the example you’ll never need. Each and every error that vgill/etc have pointed out was spot on, so I don’t need to repeat what they’ve already said.

        As for “people who have a stake in the Internet”, I’d certainly hope that would include everyone. Yes we do have some similar opinions, they stem from the the “we all have extensive experience running large multi-national networks which deliver many hundreds of gigabits per second of traffic via peering” club, and thus we have some idea of what we’re talking about. If you really can’t find the clearly identified points where we’ve collectively proven you have no clue what you’re talking about then you fail at the English language as well as at routing.

        Share
      5. “The language is specific regarding broadband subscribers and QoS (enhanced or prioritized access)”

        I just opened up the NPRM word document and ran a search for the word “QoS” and it’s nowhere in the document. The string “Quality of Service” shows up elsewhere not in the context of paragraph 106. Paragraph 106 also doesn’t mention anything about the sale of “Assured Forwarding” or “Express Forwarding” in the context of DiffServ. It merely states that it doesn’t want ISPs charging Content/Application/Service providers for either enhanced or prioritized service.

        Furthermore, the FCC’s chief technologist Jon Peha has stated that the NPRM was intentionally vague to get as much discussion and comment as possible. Dr. Peha also stated that the FCC doesn’t really understand this stuff and they want someone to explain it to them. Since we have a lot of lawyers like Tim Wu and Lessig and Free Press saying that the restrictions against ISPs aren’t stringent enough in the current NPRM draft, and since the FCC considers what this group of people say, it’s very reasonable to come out and argue that products like Paid Peering should not be banned.

        Now if it turns out that the FCC doesn’t intend to ban Paid Peering and they come out and say Bennett and Norton misinterpreted the vague language of the NPRM, then nothing would make Bennett and Norton happier. But they way you guys are turning this into some kind of holy war against Norton and Bennett and using some of the most inappropriate personal attacks is just ridiculous.

        If you don’t agree with their interpretation, fine. But you surely must admit that the issue needs to be cleared up by the FCC.

        Share
      6. George Ou wrote:

        ‘I just opened the the NPRM word document[...]“

        It does clearly say “enhanced or prioritized access”, which anyone with expertise can translate to traffic shaping, queue juggling and the passel of techniques we in IP networking call “Quality of Service”. Unlike ‘professional bloggers’ I don’t suffer from logorrhoea and prefer economy of words, therefroe used the well-known acronym QoS to cover the entire rubric. Next time I’ll show each steps so you can keep up.

        While you have the document open, please read the remainder of the sentence ” as illustrated in the diagram
        below.” and refer to Diagram 2 on that same page. It CLEARLY points to the last mile, which is appropriate as that is generally the oversubscribed segment where mitigation of abusers to avoid harming the other users on the segement takes place.

        Further you say “If you don’t agree with their interpretation, fine. But you surely must admit that the issue needs to be cleared up by the FCC.” No. There is plain simple English and a pretty picture – it is NOT up for ‘interpretation’. While much of the NPRM is vague, this is pretty bloody precise. To date, no inter-provider peering takes place on this segment so until 106 or the diagram changes, or 107 or another section is deemed as abl to trump this VERY SPECIFIC limit on the scope, it is a waste of time.

        Share
      7. JZP says: “It does clearly say “enhanced or prioritized access”, which anyone with expertise can translate to traffic shaping, queue juggling and the passel of techniques we in IP networking call “Quality of Service”.”

        It never says “Quality of Service” in regards to the FCC prohibiting ISPs from selling access to content/app/service providers. Yet the FCC was very clear in multiple places to use the full term “Quality of Service” in other places in the document where it wanted to explicitly refer to any kind of Priority Queuing technology. It was notably vague here in paragraph 106.

        And as Richard Bennett mentioned, those of us who are familiar with some of the internal affairs at the FCC have some good reason to fear that “Enhanced and Prioritized” does apply Paid Peering.

        JZP says: “It CLEARLY points to the last mile, which is appropriate as that is generally the oversubscribed segment where mitigation of abusers to avoid harming the other users on the segement takes place.”

        This obsession with the last-mile is pretty ludicrous. It makes little difference whether you apply pressure on the second hop or a little further upstream in terms of the shaping on the traffic and you know it. There are all kinds of way to shape traffic and Priority Queuing is only one of the mechanisms and probably less used mechanisms compared to the practice of just buying lower contention higher exclusivity circuits. Just having that private 10 Gbps Paid Peering connection directly into an ISP gives your traffic a hell of a boost over other traffic.

        The last mile does offer some congestion challenges but these are largely intra-subscriber issues rather than inter-subscriber issues and that makes a huge difference. The broadband subscriber has quite a bit of control in intra-subscriber congestion just based on the active applications they use but they have very little control over the inter-subscriber congestion issues and they have little control over how congested or unreliable a long-haul transit connection gets.

        JZP says: “Further you say ‘If you don’t agree with their interpretation, fine. But you surely must admit that the issue needs to be cleared up by the FCC.’ No. There is plain simple English and a pretty picture – it is NOT up for ‘interpretation’.”

        Your response is surprisingly unreasonable, childish, and tyrannical. It’s one thing to say “your interpretation is wrong”, it’s another to just say “shut up, you have no right to ask for clarification”. This is clearly against the spirit of the FCC’s NPRM when the FCC’s chief technologist says they need to figure this stuff out and they want dialog.

        JZP says: “While much of the NPRM is vague, this (paragraph 106) is pretty bloody precise”

        Actually, that’s the problem. The NPRM is very explicit in most parts to specifically use the term “Quality of Service”. It was notably missing and notably vague in paragraph 106.

        Share
    2. “Look, I think it’s really noble that Vijay has come in here guns blazing to defend the honor of his employer, and that his friends are standing up for him, but running the discussion down a rat hole of terminology isn’t really advancing the discussion or helping to move the US toward a good Internet policy framework.”

      Richard, I think what people are saying and you are demonstrating that you do not have a clue as to how things work. The basic point people are bringing up is that you are completely unencumbered by facts, so your opining about things is considered harmful.

      Share
      1. Well Vijay, what I see is you and your buddies spreading a fog of disinformation. Arbor Networks did a two year study of Internet traffic trends and found a shift away from transit and toward paid peering. This has implications for some of the proposed regulations for the Internet, under a common interpretation of the regulations. This impact happens to benefit Google.

        So you come along and very aggressively start calling names a questioning the competence of people who had nothing to do with the study for reasons of your own, and then you obviously enlist some of your friends to help you out because you can’t turn the discussion around by yourself. So instead of talking about Internet traffic and regulation, we’re now talking about who’s entitled to have an opinion on the subject.

        Let’s face it, Vijay, you don’t like a wide-ranging, free-wheeling debate, you want to spin Google’s policy preferences. I understand why you want to do that – you’re a loyal employee and you have your stock options – but your tactics are a dubious. If you have a point to make, or some data to offer, bring it on, but your argument that anyone who disagrees with you is necessarily clueless is not really working for you.

        Share
      2. Right Vijay, your first comment was to call Norton and me clueless, and you’ve simply repeated the charge several times; and you’ve asked your friends for help.

        I think everyone reading this far down knows this is your argument: Norton and Bennett are clueless, so how dare they criticize Google, the Not Evil Company. But the action here kinda belies your claim; it’s pretty evil to bully and pile on, even to protect your stock options, but that’s who you are and we all know that about you, so no big deal.

        Forget about me and forget about Norton, just go read the Arbor Networks study. Are they right that there’s been a shift in Internet traffic dynamics to accommodate the rise of video streaming? Does this have implications for ISP business plans? Does the net neutrality agenda need to protect the ability of ISPs to deal with the traffic that YouTube is dumping on them?

        I’m happy for people to consider those questions, and to do it in their own time away from all the tantrums, name-calling, and bullying. If they do that, I win.

        Share
      3. you didn’t even know what the difference between on-net routes and paid peering +was+ richard. you didn’t know. The arbor study showed a shift from tier-1 traffic patterns to direct connection. How do you know that it was paid peering? Paid peering and direct interconnection are indistinguishable at the technical level.
        I would have a debate but since you don’t even _understand_ the problem, I am at a loss on how to proceed. Its like saying ‘the earth is round’ and the answer is ‘I like french fries.’ And then claiming that there is no reasoned debate. Richard, you can’t get there from here.

        Share
      4. Google’s Vijay Gill asks: “The arbor study showed a shift from tier-1 traffic patterns to direct connection. How do you know that it was paid peering? Paid peering and direct interconnection are indistinguishable at the technical level.”

        I think the shift is mainly to paid peering since the smaller networks don’t generally qualify for settlement-free peering as they don’t have the size and scope of the larger networks. While most interconnect agreements are made under NDA and therefore aren’t public, there are a few operators – Comcast is one example – who publish their criteria for settlement-free peering, and from these we can deduce that there’s likely been an increase in paid peering.

        So that’s the basis for the reasoning. Any questions?

        Share
      5. Richard Steenbergen Monday, November 23, 2009

        Richard,

        I think you’re misunderstanding the reason for the number of comments on your post. We aren’t defending Vijay because we’re his friends, we’re defending him because he’s right. Vijay and I (and Patrick, and Dan, and Joe, and many others who have commented here) do not agree with each other on a great many issues, most of us don’t work for Google, and many of us come from very diverse backgrounds with very different philosophies on peering (both personally and professionally). On any other day you would find us all arguing with each other, but your post is so wrong on so many fundamental levels that it has brought us all together to try and help explain where you went wrong.

        I realize you’re probably just going to dismiss this as another part of the grand conspiracy against you and your ideas, but try to consider the facts here for a moment. You have a literal army of people who have many many years of ACTUAL EXPERIENCE building and managing some of the largest peering networks in the world, banging at your door collectively telling you that you are confused on even the most basic concepts of this subject matter.

        This is like watching someone argue with the top 10 heart surgeons in the world because they’ve been watching medical dramas on TV for the last 20 years and think that equals experience. You simply don’t have a prayer of knowing what you’re talking about no matter how well intentioned you may be. We’re not trying to be mean here, but you are being so resistant to clue that you’ve left us with no other options. If you have any interest in actually learning the subject matter so you can stop embarrassing yourself, you’ve actually managed to collect the right people for the job. If not, just tell us that you’re refusing to listen to anyone who might know better so we can get back to doing better things with our time. The choice is yours.

        Share
      6. @Steenbergen: I know for a fact that Gillmore, Temkin, et. al. are Vijay’s buddies and that they work for companies with a stake in the Internet. It’s not a “conspiracy”, it’s just a collection of NANOG buddies who happen to think a certain way.

        The only “factual error” that Vijay has pointed out in my post is something that he interpreted in a unique way: I said that the use of paid peering is a “new wrinkle”, and he took that to mean that I was saying that paid peering itself is a new wrinkle. That’s the extent of the so-called factual errors.

        What’s happening here is a typical Internet pile-on by a group of people with technicians disease who are going to be terribly embarrassed by the comments they’re left here in a few hours; well, that and the typical case of people trying to protect their stock options by stifling innovation. We see that all the time, so it’s no big deal.

        Share
      7. Be specific, Steenbergen: point to an actual error in my post instead of waving your hands. The only “error” that Gill has been able to find is my calling paid peering a “new wrinkle” because it’s much more widely used now than in the past.

        Calling thing of this sort “errors’ indicates a very poor level of analytical reasoning on the part of the critic.

        Share
      8. That’s basically just a bunch of hand-waving, Steenbergen. The only specific factual issue anyone of Vijay’s buddies has raised to far is the assertion that the wide-scale use of paid peering “isn’t terribly new;” that was from the anonymous JZP. Others simply declare me unentitled to opine, for no particular reason.

        The Arbor study is evidence that traffic is shifting, and the carrier-neutral peering site managers I’ve spoken with tell me they’re making something like 300 cross-connects a month. Do you think all those cross-connnects are implementing settlement-free peering or conventional transit agreements? I’m surmising that they aren’t.

        If you have some data on this, then please share, but if you’re just here to play Friends of Google, then keep on mumbling.

        Share
  15. Richard Bennett once claimed to have invented blogging. He pulled that claim from his own blog — which he humbly named “The Original Blog” — after several people showed it to be untrue.

    Now he’s telling us he’s among a cadre of elite engineers who understand the Internet better than anyone else. And yet this article misses so many basic points about network engineering to cast serious doubt on that claim, too.

    At least his banter was believable to the folks at ITIF, who hired him — courtesy of their phone company sponsors — to spread doubt about efforts in Congress and now at the FCC. Om seems to have fallen for it as well, alas.

    This is as much as Reuters could uncover about ITIF:

    “The group gets some funding from industry, including International Business Machines Corp and Cisco Systems Inc, and also is supported by telecommunications companies.”

    http://www.reuters.com/article/marketsNews/idUSN0749664720090107

    Simply pointing out that you are being paid by companies that have a stake in the outcome of the NPRM is not resorting to “personal attacks,” Richard. It is important to know who we’re talking to and who is standing behind him with a paycheck and a pat on the back.

    Share
    1. Another scurrilous personal attack from someone who doesn’t want to use a real name.

      I never claimed to have invented blogging, Kathy Marie, I claimed to have a web site in 1996 that looked kinda like a blog in that it was in newest first log format. Sometimes people do something called “tongue in cheek” commentary; you can look that up.

      Share
  16. What I’m really shocked by is the lack of full disclosure on the part of Bill Norton.

    Bill Norton is the Chair of the “Interstream Association”, which is a shell organization set up by Numetra, a company that is attempting to pitch a paid/settlement based “peering” system by which content owners, content providers, content distribution networks and their intermediary networks get paid for delivering content to eyeballs.

    Bill has appeared to fail to disclose his association to Numetra via Interstream while attempting to sound like an authority on the subject of free and paid peering. What hasn’t been said is that classic free and paid peering are not in the best interests of Numetra. Having Bill tell us that paid peering is bad is like having a coal miner tell you that hydrogen power is the worst idea ever.

    In the interest of my own full disclosure, I work at Netflix, however I am not speaking on behalf of my employer. What I will say is that the model that Bill says is broken, “the best effort delivery model” has worked incredibly well and will continue to work well for those that design their technology to adapt to changing network conditions instead of assuming that the wire will always be clear and unchanging.

    Share
    1. Dave, you’d do better to stuff the personal attacks and simply put your arguments out there. There are a number of comments here by people (Patrick Gillmore, for one) who have skin in the game and haven’t disclosed their affiliation. I can tell you you’re wrong about Bill’s connection to Interstream, and I’ll leave it to the relevant parties to correct your misunderstanding.

      What we have on the Internet is an Innovator’s Dilemma. Some companies – Google, Akamai, and Netflix among them – have invested an enormous amount of money to create the ability to accelerate the delivery of video streams on a network built on the best-efforts model that the Internet has traditionally used. These businesses are uncomfortable about changes in the ISP/Internet core service model that might open a door to smaller companies who haven’t made such an investment and aren’t likely to make one in the future. So the status quo companies are arguing for a set of regulations that will protect them from competition. It’s understandable that they should do that – industries always try to use the regulatory apparatus to protect themselves from competition – but it’s not necessarily the best policy choice for the US as a whole.

      So by all means, argue for your employer’s interests, I expect nothing else, but while you’re doing this, some of us will continue to explore the technical and policy space for better alternatives.

      And thanks again for the lunch, these ideas about paid peering were actually formulated toward the end of it.

      Share
      1. No, actually, I’ve sat through the entire Interstream/Numetra presentation and I can tell you that I feel I’m correct about his affiliation.

        Why is Numetra the only member of Interstream?

        Why does this entire post read like a PowerPoint deck from Numetra?

        Hm.

        Share
      2. “And thanks again for the lunch, these ideas about paid peering were actually formulated toward the end of it.”

        Actually, no, WBN at least has been talking about this concept for awhile now. WBN hosted a BOF at the last NANOG called “Federated CDN” which was little more than a market intelligence event for Numetra for which he was admonished. There was a reason why the talk was rejected by the program committee.

        Share
    2. Dave,

      My name is Scott Landman and I am the Vice President, Business Development for nuMetra. Over the last two years, I have been part of a global effort to reach out to carriers, content providers, aggregators, network equipment and consumer electronics firms. As part of that effort we have spoken with a large number of people and companies, many of whom believe in our open and transparent approach to building consensus and infrastructure to provide network services that meet the expectations of consumers and the needs of applications.

      The ad hoc BOF at the May NANOG was part of the global outreach. As I had indicated to you after the event, we were grateful that you were able to attend and share your viewpoints with us.

      However, I do not think that it is appropriate for you to claim that you understand the breadth and complexity of the InterStream model or have any special insight into what our motivations are based upon your attendance at a single event where only a small slice of what we were doing was discussed.

      Just to set the record straight, InterStream is completely agnostic to business models. As long as there is some sort of premium traffic with some sort of associated revenue flow, then the InterStream model can be successful. Your statement that free and paid peering are not in the best interests of nuMetra simply isn’t the case.

      Regarding your question about who else is part of InterStream, you are absolutely correct that the effort is being led by nuMetra. When building an association, typically, there is a single company or a small group that promote the idea ahead of more mainstream involvement. InterStream is no different in this regard. Further, you are correct in that we are the founders and current funders of the Association. However, that is very typical at this stage of consensus building and not an indication that InterStream is a shell set up by us for some unstated purpose. The purpose of having an Association is all over our wiki and web site. We aren’t hiding anything…

      Next, no one pays nuMetra to put a box into their network. The code that we are providing to carriers, network equipment, and consumer electronics manufacturers is done so at no cost.

      Because of the current political environment – this blog being a perfect example – we need to act as a trusted agent and in a way that allows folks from all sides of the debate to come together in a neutral and safe environment where real issues can be worked out in a flame-free environment. As such, we don’t advertise who is engaged with us at this point. Of course, when the Association is formalized it will be open and transparent.

      If you truly do want to better understand InterStream then, please, feel free to reach out to me at scott at numetra dot com. I would be more than happy to share with you the same vision, current status, and next steps that we share with everyone else.

      However, if you prefer not to engage with us then, please, do not speak for us, cast doubt on what we are doing, or make vague comments like things smell funny until you have all the facts.

      Share
      1. There is a big difference between “engaged” and “signed”. You can get a meeting with anyone. Whether or not they’ll buy off on it is a different story. Calling yourself a “Trusted Agent” is utter bullshit because you have a vested business interest in breaking the current peering model. In fact, your entire business model is based on it.

        I’ve had the vision shared with me – both InterStream and nuMetra – via PowerPoint decks showed to me by Jeff Turner and Bill Norton (and a few other people; I forget who else was in the room from nuMetra). I obviously disagree with your vision, and will retain the right to comment on it – I kept silent until the “ad hoc” BOF which busted it out into the open.

        InterStream, by Jeff Turner’s account, has been around for many years. Why is nuMetra only now claiming responsibility for it? You guys seriously need a lesson in how to achieve “stealth mode”.

        This is my last post on the InterStream/nuMetra matter as this is becoming circuitous, but rest assured I have seen the same vision as everyone else, unless a different vision was presented to me, and I personally feel it will be a failure and actually open the door for a whole slew of other ‘bad things’.

        Share
  17. What I’m really shocked by is the lack of full disclosure on the part of Bill Norton.

    Bill Norton is the Chair of the “Interstream Association”, which is a shell organization set up by Numetra, a company that is attempting to pitch a paid/settlement based “peering” system by which content owners, content providers, content distribution networks and their intermediary networks get paid for delivering content to eyeballs.

    Bill has appeared to fail to disclose his association to Numetra via Interstream while attempting to sound like an authority on the subject of free and paid peering. What hasn’t been said is that classic free and paid peering are not in the best interests of Numetra. Having Bill tell us that paid peering is bad is like having a coal miner tell you that hydrogen power is the worst idea ever.

    In the interest of my own full disclosure, I work at Netflix, however I am not speaking on behalf of my employer. What I will say is that the model that Bill says is broken, “the best effort delivery model” has worked incredibly well and will continue to work well for those that design their technology to adapt to changing network conditions instead of assuming that the wire will always be clear and unchanging.

    Share
    1. Given that Bill Norton didn’t write the post or leave a comment, how on earth is he supposed to disclose any employment relationships he has, Dave, by ESP? Akamai’s guy, who has posted some quite nasty comments here, hasn’t disclosed his employer, and several of the other NANOG buddies are too cowardly to even post their names, let alone their employers.

      The point of the exercise is to have a discussion about the issues, and we were doing fine until a pompous little blowhard came waddling in here doing what he does best. If you have something to say on the issues, fine, say it, but leave the character assassination at the door, please.

      Now you do finally get down to something close to an issue after you beat up on Bill, namely the question of best-efforts delivery which has been close to a standard for Internet transit for a while now. Your assertion, that people should design around it, is the principle that has motivated the design of the Google and Akamai networks. That’s fine, they’ve dumped $40B into them to accellerate their apps and they feel content.

      The trouble is that every new entrant shouldn’t be forced to do business with Google, Amazon, or Akamai in order to have a chance to compete, and the Internet shouldn’t be held hostage to apps that can only work on a best-efforts network. Protecting the status quo is not as important as supporting innovation with new apps and new business models that aren’t practical on an Internet that’s controlled by a handful of CDNs and overlay networks. That’s the actual issue.

      Share
      1. First, I am only speaking for myself – in this capacity I am working for the Internet, so you can get off the high horse. The errors in your analysis are many, starting with the assertion that ‘paid peering is new’ When called on it, you said you were not sure if on-net routes and paid peering was the same, relying on bill nortons paper, which had been thoroughly corrected by a wide variety of folks who were all more qualified to speak on the matter. Then you moved the argument to the shifting of traffic flow – yes, it has shifted because other peoples networks are now bigger than they were before and what used to be a hierarchy is now flattening out. The fact that it is paid peering or not has no bearing on anything, especially since it is a purely commercial transaction. The fact that you can’t even comprehend such basic facts leads people to try and point those issues out, which are then interpreted as attacks. Then you start complaining about 40+ billion spent on networks? Where are you getting those numbers from? The net net is that there is simply no point in continuing this. As for holding the internet hostage to apps that can only work on best effort basis, well, it appears to have worked fairly well so far. This reminds me of the ATM to the desktop folks – “but 25mbs to the desktop is going to beat 100mb ethernet because we are BETTER”

        There is no conspiracy and friends recruitment, in fact it would be hard to find a point that patrick, steenbergen, jzp, dan golding etc all agree on. That such a wide variety of folks have come together speaks volumes about your inaccuracies.

        /vijay

        Share
      2. Your entire post is based on (completely untrue) assumptions from Bill’s paper. Of course it’s directly relevant; stop deflecting.

        Share
      3. Adding additional paths to a destination (more aggregate capacity/diversity) does not negatively impact competition. It generally reduces overall utilization… better performance for all.

        The barrier to entry for Internet connectivity is considerably lower than 3 years ago. How many of today’s business models would float with bandwidth at 2005 prices? 2006? maybe 2007? Today… anything is possible! I see many new faces at conferences, new products, new markets. There are more entities controlling the internet… not less.

        I’m sorry a “pompous little blowhard” peed in your cheerios, but that doesn’t change the fact that your post is littered with implied conspiracy + drama where there really isn’t any…

        Share
      4. Adding additional paths to a destination does not negatively impact competition. It generally reduces overall utilization… better performance for all.

        The barrier to entry for Internet connectivity is considerably lower than 3 years ago. How many of today’s business models would float with bandwidth at 2005 prices? 2006? maybe 2007? Today… anything is possible! I see many new faces at conferences, new products, new markets. There are more entities controlling the internet… not less.

        I’m sorry a “pompous little blowhard” peed in your cheerios, but that doesn’t change the fact that your post is littered with implied conspiracy + drama where there really isn’t any…

        Share
    2. As CEO of nuMetra, I want to shed factual light on the comment made that InterStream is a “shell organization set up by nuMetra.”

      Firstly, for those unfamiliar with InterStream:
      “Established in 2002, the InterStream Association is a multi-industry organization responsible for creating, enabling and promoting a fair and transparent global Internet infrastructure which secures delivery of licensed content to broadband-connected consumers. The Association seeks to address the lack of an industry-wide infrastructure for mediating (i.e., segmenting) traffic according to the needs of applications for broadband-connected consumers.”

      http://interstream.com/about

      To separate fact from fiction, in 2004, the thought leaders who originated the concept of the InterStream Association recognized the importance of separating standards body functions from validating network integrity (through bandwidth and latency measures to ensure consumers get the bandwidth and latency they are promised and pay for). Thus, nuMetra was formed (from http://numetra.com) “as technical, ‘end-to-end’ clearinghouse solution for the entire backbone of the Internet that can be implemented by both wholesale and retail ISPs.”

      Through a cooperative rather than exclusive approach, InterStream wants to minimize inherent conflicts of interest such as those existing with text web, ‘kings of http’ (i.e., lack of transparency surrounding CPC text ad fraud with little incentive for 3rd party audits to ensure CPC text ad customers ‘get the value they are being promised’) in order to maximize the value of the entire Internet value chain.

      The members of InterStream serve as a critical ‘check and balance’ as they can remove nuMetra in its administrative role if nuMetra does not act in a fair and neutral matter. Furthermore, it is anticipated that several entities will end up offering competing administrative services once commercially live (a good analogy is the evolution of the credit card processing industry; http://www.creditcards.com/credit-card-news/credit-cards-history-1264.php ). By default, nuMetra is simply taking the lead in order to productively ‘move the ball forward.’

      According to comScore, the median U.S. consumer is currently receiving 50% of the bandwidth they are being promised (IOW, paying 50% more than they should be). [Source: FCC website, http://tinyurl.com/yl68x8g%5D

      According to Pew Internet, in 2009, only 44% of U.S. households with income <$30k subscribe to broadband versus 68% with household income between $30-75k. [Source: Georgetown U.-sponsored study, http://www.gcbpp.org/files/Academic_Papers/AP_Hassett_Shapiro_Towards.pdf%5D.

      Simply put, lower retail broadband cost = higher number of broadband households = more potential Netflix customers & more legitimate CPC text ads clicked.

      Anonymous, given the aforementioned facts under the backdrop of today’s ‘Main St.’ recession alone, I do not believe a strictly ‘best-effort’ model is in the long-term best interest of any part of the Internet value chain, let alone cash-strapped consumers.

      Share
      1. nuMetra CEO,

        As someone hosting content, this thread naturally intrigues me.

        Would you be able to shed some more light on what measurement criteria you’ve used, or who you’ve spoken to, to ultimately come to the conclusion that a “best-effort” Internet is inappropriate for serving content (be it static images, video, audio, …)?

        This concern has never come up with our customers, not even once.

        (I asked this question of wbn prior and couldn’t get a straight answer, just some “well, one guy in a film studio said…” line.)

        Share
      2. Mr. CEO,

        If Interstream is not a shell organization set up by Numetra, who else are members? The site has a “members page”, but lists no one. The only Google search for Interstream members turns up Numetra.

        Why is the only contact on Interstream’s website Numetra’s PR consultant, Dorrit?

        Something smells funny.

        Share
  18. Patrick W. Gilmore Monday, November 23, 2009

    Richard,

    I take great umbrage at you insinuating that I am defending Vijay because we are “buddies”. Considering half your responses are essentially complaints that everyone is hurling personal insults at you, perhaps you could respond to the facts instead of doing that which you detest.

    Also, please note that I have not said one thing about Google’s stance, the FCC’s rule, or other political topics, I am saying your -factual- assertions are incorrect. This is not an opinions or guesses about what someone or some company plans to do. I am saying you are spouting bad information about how the Internet works, so it is nearly impossible to make good decisions or predictions about what is to come.

    Worse, you claim some type of expertise based on completely unrelated fields. You explicitly disavow knowledge of certain critical aspects of the subject matter despite listing job titles which pretty much require such experience in your resume. You get some of the most basic ideas and facts regarding this topic blatantly wrong. And yet you still feel comfortable, without knowing who we are or what we do, to critique our ability to understand the same topic.

    The people responding to you are genuine experts in this field. Our experience and knowledge encompasses taking multiple networks to actual tier one status (as opposed to what Mr. Norton even claims to have done); presenting to politicians on issues of policy; writing our companies’ peering policies and other business documents; being published in peer-reviewed journals on the subject at hand; setting up 1000s of real peering sessions, private, public, settlement free, paid, and otherwise; and many other aspects of growing, running, and understanding networks.

    Despite our similarities, our disagreements are legion. One of us has espoused on more than one occasion the idea very large providers should only peer with other very large providers and everyone else should buy transit or paid peering. Another of us has the most open peering policy on the planet earth. And somehow you believe we are backing each other up because of some grand conspiracy.

    Richard is right, you do not even have the basic requirements to understand why you are wrong.

    We have tried. Most of us have been polite and even patient. You are not just difficult to educate, you are actively hostile to the idea you need to be educated. This type of intentional ignorance is worse than useless, because you are helping to propagate false-to-fact information in a forum where people will assume you are correct. This makes more work for those of us who do have to run networks, and we have no choice but to respond in kind so as to correct this misinformation.

    I am sorry, Richard. It would have been quicker, easier, and better for all of us if you could have simply admitted that maybe, just maybe, someone else might know something you do not.


    TTFN,
    patrick

    Share
    1. I suggest you read a paper entitled “Complexity of Internet Interconnections: Technology, Incentives and Implications for Policy” by P. Faratin, D. Clark, P. Gilmore, S. Bauer, A. Berger and W. Lehr. It was presented at TPRC in 2007, and you can get it at: http://people.csail.mit.edu/wlehr/Lehr-Papers_files/Clark%20Lehr%20Faratin%20Complexity%20Interconnection%20TPRC%202007.pdf

      This paper says: “This line of thought has two implications—more formal peering contracts and the emergence, as we will discuss below, of paid peering” and “The two new types of agreements we will discuss are paid peering and partial transit” and “The emergence of paid peering, although non-standardized in its details, is not surprising as a concept”.

      This suggests that paid peering was “a new wrinkle” (as of 2007) in the Internet interconnection space. Oh, BTW, the P. Gilmore who co-authored the paper would be, um, you. The paper cites Bill Norton’s research on peering prices, which kinda sorta implies he’s a bit of an authority on the subject.

      And no, “paid peering” is not the same thing as “on-net access”. Paid peering is going to take you into one and only one network, while on-net access simply gives you a closer point of presence to connect with a transit network, which is not the same thing at all. So no, I still don’t know that on-net access is the same thing as paid peering; in fact, I know that it isn’t.

      Share
  19. Richard, I’ll have you know that Bill Norton is not the peering maven he advertises himself to be.

    Last April, California revoked his license:

    Tuesday, April 14, 2009

    Dr. Peering sued for malpractice
    Earlier today, an individual who operates under the alias “Dr. Peering” was charged with peering malpractice and operating without a peering license by the California Medical Review Board. Court documents obtained by Tier 2 Research indicate that the allegations revolve around a specific set of “white papers”, which are purported to give inaccurate and misleading medical advice to up-and-coming networks on the topic of settlement-free peering.

    Tier 2 Research has performed its own independent investigation, and found several network operators who were willing comment on condition of anonymity:

    “We had a hard time getting traction in the peering space. The papers said if we build out to all these peering locations, folks would come running. It never happened and we ended up getting acquired by another CDN network. At least we don’t have that damned Panther on our web page anymore”.

    “We’re a pretty big video on the Internet provider and we thought being in Equinix was a good idea. However, we got sent to Lundy and our transport options were limited. I even bitched about it at the Peering BOF. Doesn’t matter now, ’cause I’m rich bitch”.

    “We thought being at Internet Exchanges would be a good thing. Not to peer, but to sell peering routes. After Dr Peering showed us how to skim PeeringDB, I called everyone I could trying to sell the paid peer. Its worked out pretty well, except people keep asking for this bgp community thing.”

    Attempts to contact the alleged “Dr. Peering” for comment were not immediately successful.

    Share
  20. While I am many things, using my initials hardly makes me anonymous. If you bothered to click through to the URL on every comment, or did a whois then you’d see (well, no longer in the DDN whois so you’d have to look at JZP-ARIN). I’m not cowardly, not employed by any content house, telco/cableco/ISP, have NO soapbox and at no point have thrown any personal attacks at you nor Bill.

    I’m someone who has raised more than one point, all of which you keep ignoring. In your response to Dave, you do finally admit that you are anti-best-effort packet-switched networks, which indirectly answers my question “Do you wish to regulate queuing disciplines and TCP implementations”.

    I was only stifled by telco regulations and cost constraints when building a regional ISP in 1994; overcame it. I was stifled by marketing race-to-the-bottom all you can eat with no investment to build infrastructure in the dot-bomb; managed it. I was stifled by cutting staffing while leading the pack on speeds in residential broadband built on our own facilities. I was never stifled by best effort traffic delivery, and the only ones who were are the ones who missed the clue.

    Voice was supposed to be impossible on a best effort network. Video was too. Real-time data was supposed to require lots and lots of admin overhead. All these things have proven to be false and that a level playing field promotes innovation in packet handling. No one who wishes new entrants should be interested in added regulation which apply network-wide for certain classes of service providers in the US; in the future you seem to desire [any broadband provide regulated so heavily they couldn't offer any other services or be dictated from whom to by] I see a direct analog to the way [well-intended] LATA restrictions dictated braindead network architectures on telco DSL. Of course once codified and tarriffed, some folks will find loopholes, just as all independent ISPs did in the early 1990s… then you’ve wasted time and money building and running a regulatory structure that doesn’t do what you’ve intended. To assume that people won’t do this ignores history, pigeonholes the ‘innovators’ you wish to promote, and is the height of hubris.

    I’ll ask you, what fairness doctrine will you apply in your non best-effort world? How will a new entrant get their application accepted by the QoS police? I’msure it involves lots of legal billable hours, forms, state and federal fees, etc. How is best-effort “holding hostage”? How o you make an open guarenteed system for packet-switched networks that works end to end? There could be a shred of sanity in the argument if you were sticking to the last mile, but you’ve clearly said that you don’t think fairness can be enforced without doing so across the entire netwok for a given business entity. If you really want new entrants to have a level playing field, go after bogus method patents which apply to simple things. Even though Bilksy seems to have moved us in that direction, I doubt you will undermine your own patent age on the resume.

    You assert that 106 applies to paid peering, then backpedal to say “if it is changed to say so”. Of course if the rule-making is changed it can say anything. You are not addressing any points raised other than minutae [Bill Norton's credentials, how old is paid peering, etc] and skip the myriad of other factual and conceptual errors.

    I regret to having helped drive traffic to this site if you remain so clue-resistant.

    Share
    1. A ha, JZP!

      “How will a new entrant get their application accepted by the QoS police? I’msure it involves lots of legal billable hours, forms, state and federal fees, etc. How is best-effort “holding hostage”? How o you make an open guarenteed system for packet-switched networks that works end to end?”

      Funny you should ask, and I’ll tell you how. By joining the Interstream association and paying Numetra to stick a box on your network you’ll let Numetra do that! Don’t worry, it’s fair and open because you’ve also joined the Interstream association (founded and funded by Numetra). It’s kind of like magic underpants, only it’s harder to explain.

      Share
  21. I have one more thing to complain about this already fully debunked piece of drivel. (Yes I know this is a very negative beginning, I just couldn’t find kinder words)

    You make a remark about the MINTS traffic growth numbers as being too low and therefore inaccurate. Do you have anything to back this claim up?

    Share
  22. I’m a bit saddened to see so many talented, clueful people wasting so much time on this “discussion”. It’s clear that Bennett doesn’t understand much about network interconnection and is willfully resisting polite attempts to reduce his ignorance. It’s telling that there is not a single comment in this thread (from anyone other than Bennett himself) supporting his bizarre interpretation of facts. This speaks volumes.

    Quick summary for the impatient:
    –The Arbor report says nothing about paid peering. If you think it does, you either didn’t read it or didn’t understand what you read.
    –Nothing in the FCC proposal says anything at all about interconnection. It’s about QoS, and specifically QoS on the last mile.
    –There is no difference between paid peering and selling transit for on-net routes only. The fact that Bennett (and apparently Norton, which surprised me) don’t understand this is a bit sad.
    –People from a variety of different positions on the Net Neutrality issue all agree to these facts. Bennett and Norton do not.

    Facts are hard. Let’s go shopping. :-)

    TL;DR: Bennett is wrong. Norton is wrong. The FCC rules may be wrong, but they don’t prohibit paid peering. And no one (ok, no one paying much attention) thinks they do.

    Share
    1. Todd:

      Attacking one another here serves no real purpose other than to fortify individuals into their own irrational positions. The FCC is seeking comment and I think Bill and Richard have done a great service in pointing out, depending on your point of view, the ambiguous language in the NPRM and how it MAY impact peering and transit.

      Note, from the FCC NPRM Section/Question 106:
      ‘We understand the term “nondiscriminatory” to mean that a broadband Internet access
      service provider may not charge a content, application, or service provider for enhanced or prioritized
      access to the subscribers of the broadband Internet access service provider, … We propose that this rule would not prevent a broadband Internet access service provider from
      charging subscribers different prices for different services. We seek comment on each of these proposals.
      We also seek comment on whether the specific language of this draft rule best serves the public interest.’

      Whether you believe this proposed rule would regulate interconnection (i.e. peering & transit) or not, is really not important. It is more important that you provide your comments to the FCC. The proposal clearly states that they “seek comment.” Therefore, I would suggest that all of the folks contributing to the debate on this blog redirect their energy toward telling the FCC what they think it means and how best to codify rules or principles to prevent “non-discrimination.” The vitriol going back and forth here on this blog posting is going to be of little relevance at the end of the day. Commentary to the FCC will be.

      Share
      1. Jeff,

        Todd, myself, and the other posters are attempting to prove important points:

        1) That Richard has no basis in fact for his opinions or statements

        2) That the study that Bill did that is the core backbone of Richard’s opinions and statements is fatally flawed

        3) That the people who have contributed to the information given above have ulterior, commercial motives that they are not disclosing

        It is important that all of these be attached to this blog posting, lest the blog posting be found by someone who actually has some power at the FCC and views it as a statement of fact when it’s anything but. This is the beauty of Web 2.0, it’s a lot harder to spout missives and possible untruths without having others call you out on it.

        Share
  23. [...] Read the rest of this post on the original site Tagged: Internet, Voices, digital, economy, hardware, innovation, media, video, Arbor Networks, Comcast, GigaOm, Google, Richard Bennett, Tier-1 | permalink Sphere.Inline.search("", "http://voices.allthingsd.com/20091124/how-video-is-changing-the-internet/&quot;); « Previous Post Next Post » ord=Math.random()*10000000000000000; document.write(''); [...]

    Share
  24. Sylvie LaPerrière Tuesday, November 24, 2009

    Greetings,

    I am the real Sylvie LaPerrière, the real peering director at Tata Communications. I am appalled that someone would think appropriate/funny to usurpate my identity and post any comment that would be attributed to me on thread.

    Seriously, stop this.
    Regards,
    Sylvie

    Share
    1. Sylvie,

      Thanks for stepping into this discussion. The Wikipedia folks have been having major problems with somebody defacing the “Tier 1 Carrier” article and repeatedly adding Tata to the list. Think you might be able to help out here?

      Share
    2. Sylvie LaPerrière Wednesday, November 25, 2009

      Johnny Canuck,

      If you would like an information session on the wiki page, please email me at sylvie.laperriere@tatacommunications.com

      Regards,

      Share
    3. Bob "Elvis" Gratton Wednesday, November 25, 2009

      Oh Sylvie, koi de neuf!

      These mechants on the blog, tabarnak stie. They do not understand who you are. They dont comprehend the newfound tier 1 status sur le internet. Maybe if they read the wikipedia page web they would see the truth, colisse!

      I sympathize that it must be hard to deal with the new management techniques, I ecoute les youtubes and I found this internal video of one business meeting:

      http://www.youtube.com/watch?v=Jat7Z24ZUQs

      pov fille!

      Desoler ma poupounette. Put some Plume Latraverse on and he will make the gris nuage go parti! :)

      -Elvis Gratton, les king des kings

      Share
  25. This is like a good old-fashioned Usenet flame war, only without the wit; not terribly informative, but there’s a lot of ego on display. I’ll just make a few closing observations before moving on.

    • Nobody has offered any information to show that the article is flawed in any serious way. It is true, of course, that paid peering is not new; AOL offered a paid peering product back in the days when Vijay Gill and Daniel Golding worked there, but it wasn’t terribly popular because it was priced higher than transit. The reason they priced it that way was that it provided better performance to/from AOL than transit did. That was the kind of service the FCC regulations are meant to ban. The Comcast-style paid peering which is cheaper than transit raises interesting issues for the FCC because of the pricing. The intent of the regulations – and this is something you won’t get from going to NANOG meetings – is to prevent ISPs from extracting “monopoly rents” from content and service providers seeking access to the ISP’s customers, but offering higher performance at lower price to people who have built networks that can take advantage of it is outside the set of expectations that the proponents of an anti-discrimination rule have in mind. It is for this reason, among others, that I’m opposed to the anti-discrimination rule that the FCC is considering. The consequences of an overly-broad ban in differentiated access services are likely to fall just as strongly on legitimate as illegitimate services. My article didn’t mention anything about “on-net routes” BTW.

    • Google is and has been the primary moneybags behind the drive to regulate the Internet according to net neutrality principles, and has always claimed it was promoting the program in the interests of smaller players, not for itself. The likely effects of a paid peering ban on smaller players shows that regardless of Google’s intentions, the expansive ban in differentiated access to ISP networks is not good for smaller players. And the nastiest comments in this section come from people who work for Google and claim to be protecting the Internet from ISP shenanigans. Draw your own conclusions.

    • Disclosure is a tricky thing. I don’t have a financial interest in any ISP, carrier, content network, or network services provider, I’m simply a former network engineer working for a Washington, DC think tank. The most critical comments in this section are in fact from people who work for Google, Akamai, YouTube, and Netflix and similar companies and who have, for the most part, chosen not to disclose their affiliations. You folks know who you are, so this would be a good time to ‘fess up if we’re going to play full disclosure. You might also say how you happened to hear about this article. It’s a bit unusual to see 7 NANOG members commenting on an article at GigaOm; I’ve certainly never seen it before.

    • Who has the skill to interpret FCC regulations? Given that the net neutrality NPRM is still in the discussion stage, it’s entirely appropriate to raise questions about what it might cover, and completely inappropriate to insist that the rules are already black-and-white as Vijay and Daniel have. We’re in a process of discussion, and as Jeff Turner points out the goal at this stage is to ensure that the FCC does not enact a ban on paid peering or make any similarly destructive move. I think most of us would agree that such a ban would not be beneficial for anyone except the operators of the largest CDNs (the ones who’ve been complaining the loudest about my article.) So don’t lull yourself into a false sense of security by following Daniel’s guess that the FCC would never do such a thing; they’re a government agency and they screw up and/or protect certain companies and business models over competitors all the time. The only way to prevent this is vigilance, and it’s not 100%.

    • We don’t know the value of better-than-best-efforts or cheaper-than-best-efforts transit on the public Internet because we haven’t conducted a large-scale experiment with it, but there is reason to believe that such services have value based on small-scale experiments so far. This relates to network theory and emerging business models, and isn’t in the NANOG sphere of operational expertise, so don’t worry about it. I want to see the space for experimentation with SLAs, business models, and differentiated transit and peering preserved. Obviously, some others would like to see it stopped an for a best-efforts model to be imposed on the Internet of the Future by law. That’s why we’re having a fight over net neutrality; companies that are successful on today’s Internet want to lock in the design and operation of the system and prevent it from changing. The FCC is in the middle of the fight, and these issues are international and ongoing.

    • Arbor points to the Rise of the Hyper Giants, the 30 large firms who control 30% of Internet traffic. As more traffic concentrates into fewer hands, the ability the Hyper Giants to affect everyone’s Internet experience increases. Historically, the FCC has paid more attention to ISPs than to CDNs and other streamers, but given the end-to-end nature of the Internet, this focus is inadequate to protect user experience. Going forward, if the FCC is going to regulate Internet services at all – and I don’t want them to do that – they will need to regulate all the firms who affect user experience, not just the ISPs. Once again, I would prefer they don’t impose any new regulations, but if they are going to regulate, they should apply the rules to everyone, not just to ISPs. And there is nothing in the NPRM that limits it to last-mile netorks, that’s junk; it applies to ISPs period, both on the customer-facing side and on the Internet-facing side.

    So this is where we are. The FCC is floating some rules, some people are constructing scenarios and discussing implications, others are trying to stifle discussion by various means. Do what you want, but don’t kid yourself: there’s a lot of money at stake, and the established players aren’t going to roll over and play dead without a fight.

    Share
    1. HELLO
      BLACK_MAN HERE
      Where is my operator status?

      MR. BENNETT,
      It is not NANOG that attacks you, it is the IRC (RFC 1459) community from #ix on irc.terahertz.net that attacks you. It is here where all the peering big ballers hang out, you should stop by and say HELO. Many of the #ix members blog about channel events, please stop by!

      Share
  26. Mr. Bennett,

    You don’t know me, but many NANOG folks do. They and I are colleagues, some are buddies, a few even friends. I don’t speak for them, or anyone. I can say for a fact that the folks commenting here are among the very top in their field.

    While my background in networking (including building a Tier-1 network from scratch) and non-technical aspects of operating ISPs (including regulatory issues, cost models, and peering/transit) informs me, it does not limit my thinking. And just as important, I have no skin in this game. I don’t work for a content provider, carrier, ISP, or other networking entity.

    I analyze. I think. I learn. And what I learn, I teach.

    What NANOG is about, is teaching, and sharing ideas. That the Internet exists at all, is solely due to the cooperative efforts of engineers (not technicians! – that is like calling a brain surgeon an orderly!).

    Here are some of the assertions which are not backed by evidence, and which I believe to be false. They are the backbone of the “straw man” you are attacking, and once they are pointed out, your argument will quickly fall apart. Well, not “your” argument — your employers’ and their clients’.

    (1) None of the traffic changes are “in response to video”. They are in response to aggregate traffic, and are in fact content-neutral. Content-agnostic, or even content-unaware might be better terms.

    (2) Peering links, paid or otherwise, are all best-effort.

    (3) All QoS is, is a method of handling congested “pipes”.

    (4) Any time new connections are added, they remove the need for QoS. Adding capacity is not “a poor man’s QoS”, in fact it is the antithesis of QoS.

    (5) #4 above applies to all connection types – transit, peering, paid-peering (aka “on-net”), and speed upgrades to consumers.

    (6) Applying preferential mechanisms (like QoS) has already been attempted on a wide scale previously. It had a brief period of modest success, and then died a well-deserved death. The benefits were always dwarfed by the costs, and that is a fundamental, mathematical necessity. The technology is still known by its acronym – ATM.

    (7) Question 106 is specifically referring to the concept of “selling the CONSUMER” differentiated services, i.e. passing the cost of the differentiation on to the person already paying for the access. Similarly, the question of “specialized access” refers to the last-mile consumer.

    (8) The ISP community is not concerned over the NPRM having any possible application to peering circuits, paid or otherwise. Stay off their side, or rather, stop pretending you are on their side.

    (9) Since no peering connections are using QoS, there is no worry that mandated net-neutrality would impact those connections.

    (10) The FCC is concerned over network elements covered by monopoly provider access. Last mile is such an element. Peering links are not.

    (11) The only way the FCC is likely to curtail regulation on the last mile, is if the last mile network elements, and ownership thereof, are unbundled from the networks who currently have the monopoly on those. I don’t think your employers’ clients’ are interested in going there.

    Sincerely,
    Brian Dickson

    Share
    1. That’s an interesting comment, Brian, and one that deserves a response if for no other reason than the fact that it’s mainly polite and a good illustration of what we’re up against as we try to move the Internet into the 21st century. Let me take your assertions in order:

      1) Most of the traffic increase is in fact video traffic; this is well-known.

      2) Most but not all peering links are best-efforts; I have been told by both academics and operators about limited arrangements that preserve QoS at peering interfaces. Be careful about sweeping statements.

      3) Moments of congestion are typical of every packet-switched network; by design, packet-switching seeks to use all available bandwidth.

      4) Over-provisioning only relieves congestion temporarily; any system that seeks to limit latency is a QoS system in my book.

      6) Not really.

      7) Question 106 is among a number of ways of addressing the proposed anti-discrimination rule. What it will mean at the end of the day is still up in the air. One element that the proponents of net neutrality have been very big on is preventing ISPs from selling QoS to service providers, and that’s the intent of Q. 106.

      8) Neither of us is anointed to speak for any communities as far as I can tell; do you have an employment letter?

      9) See answer 2. The issue isn’t so much what the current, 20th century practice is as what sort of space carriers and ISPs will have for experimentation in the future. It’s a mistake, in my book, to declare Internet engineering over and done with; there’s more to learn.

      10) It’s a mistake to read FCC regulations too narrowly. For several years, we assumed net neutrality would only apply to wireline ISP connections, and this fall the FCC threw us a curveball and applied them to wireless as well. It’s certainly within the realm of reason that the effort to regulate ISPs will not be limited to the customer-facing side, and in fact never it never has been limited in that way. This is very easy to discover by following the regulatory debate.

      11) not a substantial comment.

      So what we have here is fundamentally a question of whether the Internet is to be prevented from changing in any serious way, or if we have to keep stumbling along from crisis to crisis by pulling more fiber and building more CDNs.

      We’ll obviously keep doing that, but this means isn’t going to enable a tripling of the user population as the wireless world hooks up to the Internet, and brings its voice interactions along.

      Share
      1. Thanks for a point-by-point reply. I think more good-will could be found by doing a few more of those, and/or to address a few key points by Vijay and by Richard S.

        Here’s where I think the lay-persons and the policy-types actually need more education, and why it is important that everyone involved in pushing the policy envelope needs to be careful: this is rocket science.

        The essential problem is that the Internet’s traffic pattern has been exponential, for at least the last 20 years. There have been a few wobbles in the rate-of-doubling, mostly related to the effects of shifting regulatory sands (leading to, among other things, the dot-com bomb), the economy, and the odd technology-related stutters.

        The very nature of exponential growth for an extended period of time is very difficult to grasp, especially outside of the network engineering field (and even sometimes inside it).

        Case in point – the “one fiscal quarter at a time” crowd tend to try to treat each quarter as if things were linear, or to compare year-over-year. They use the wrong axes on their graphs – they should use log(10) instead of 10 for the units.

        The numbers involved are beginning to scare the bean-counters. And that is the “jitter” we should really be concerned about. Unless they are educated on both the plus and minus sides of the equation, they are likely to panic rather than “get it”.

        And where exponential growth leads to problems is the idea that costs grow as traffic grows. This leads to the idea of needing to find ways of making more margin from the existing revenue stream.

        In fact, the unit costs of Internet links are shrinking all the time. Transit prices per Mbps are shrinking proportional to the volume, i.e. roughly speaking, volume*unit-price = constant, and margins are generally preserved.

        QoS has always been a solution looking for a problem, and monopoly players + QoS is a perfect storm of brain-damage.

        The problem is, that once the QoS fuse is “lit”, it’s an end-game move. QoS does not create bandwidth, it at best manages a finite amount. It is sold to providers who are looking for a way of deferring upgrades. Once they have a solution that lets them charge for “premium” content, they can and likely will perpetually starve the non-premium content. This ignores that the non-premium content effectively paid for and built the infrastructure that the premium content now wants to hijack.

        It is a case of killing the golden goose. And at the same time, it is illusory at best.

        For example, QoS can do some good things for jitter. But, if traffic continues to grow, it hits a fibre-optic glass ceiling. Once the actual combined traffic peak exceeds capacity, a queueing-theoretical point is reached, of non-bounded queue length. Basically, for the consumer, the amount of buffering needed for high-bandwidth-content (like streaming video) grows until it becomes unusable.

        And it is surprising how quickly that can happen.

        What the monopoly crowd needs to realize is that there is more money to be made by actually adding value, by actually aggressively upgrading capacity, than there is in “choking” the usage. Bill for it, don’t stop it. Network capacity is the new “crack”, and the last mile is the “crack pipe”. Except it’s legal, and loosely regulated. How hard can it be to understand the model? It’s a pie – you don’t need a bigger piece of the pie, you make the pie bigger.

        And to address a few points in your reply…
        (3) Actually, no, congestion is very rare these days, at least at the backbone layer of networking. Network engineers have known for ages that you can only ever push the bottle-neck to somewhere else. We’ve pretty much managed to push it to the edges of the networks – the last mile, and the servers themselves, and the latter only rarely even (like flash crowds).

        (4) Everything is temporary. And everything can be viewed differently depending on time-scale. Two quotes come to mind: “Failure to plan on your part does not make it an emergency on my part.”; and “But the last thousand years have been atypical.” (The last quote is from http://www.benford-rose.com/scientistwrite.php).

        And finally:
        (2) If you dig deeper into what kinds of peering links have QoS preserved, and what the relationships on those are, I think you’ll probably find they are extremely atypical. I would expect they are primarily:
        (a) VPN traffic (especially MPLS BGP VPN, 2547bis), or (b) VoIP traffic (specifically SIP trunks), or (c) other content-specific “trunks” which do not carry general-purpose traffic along side the special content, such as multicast traffic, or SAN traffic.

        And (d), all of these peering links are uncongested, with the QoS parameters only being used for “marking” the traffic for use elsewhere in the networks of the respective peers (and not of third-party networks, since QoS parameter will get discarded at the far side of both peers’ networks). Please dig deeper on this one – if the small proportion of QoS-peer-links have a percentage of links with actual congestion on them, exceeding 0%, I am sure lots of people would be interested in knowing it – and likewise, if it is 0%, that is also a very telling data point.

        Thanks,

        Brian

        Share
      2. I addressed Vijay and Richard S.’s legitimate points already, but they continued flaming, so that’s all I can do with them.

        The QoS fuse has already been lit for broadband, even if it hasn’t been lit for the Internet as a whole. Triple-play services in residential, last-mile networks and SLAs for commercial transit already use QoS, and they have to for reasons that you’ve given. The Internet core (peering and transit) deals with congestion by pushing it to the edges, which is to say the residential networks and the commercial transit. This doesn’t mean that congestion is gone, it simply means that it’s outside the core’s radar. In order to provide voice and video services, broadband providers provision and manage traffic in such a way as to provide lower latency for voice and video than for Internet access. If we want to provide a milieu where over-the-top services can be competitive with ISP-provided triple play, there needs to be a means for these services to get the same level of QoS in the last mile that triple play already has. And it’s reasonable for this level of service to come with a fee attached. This falls out of the facts you’ve cited, so there it is.

        And this brings us to an important point about the Internet as we know it today: from the standpoint of network theory, this network is one of many possible networks, and there’s no reason to believe, a priori, that it’s the best of all possible networks. In fact, the Internet has historically served a rather narrow band of applications, namely those that aren’t terribly latency-sensitive at all (web surfing, e-mail, ftp) and those that are latency-sensitive but are self-limiting with respect to bandwidth and typically not very demanding, such as VoIP.

        In the future, two main trends are likely to hit the Internet that it’s not terribly well-engineered to handle today: a radically larger user population (as mobile users come to the Internet) and a more diverse set of applications, some that require low latency while consuming large amounts of bandwidth, such as video conferencing. In this future scenario, it’s going to be necessary to provide a different level of service for some applications than for others, especially over the wireless last mile. Hence, conventions about how to manage the Internet, how to package services, and how to manage wireless packets will have to change.

        It would be a real disaster if regulations intended to solve a non-existent problem (net neutrality is a solution in search of a problem if there ever was one) prevent the engineering from being done that will be necessary in order to extend Internet service (with VoIP and video) to the 3 billion mobile handset users who are not connected to the Internet today.

        This the problem that preoccupies me.

        Share
  27. Jeremy Penston Monday, November 30, 2009

    Better late than never – I’m a bit disconnected from the topic these days, but as the person responsible for writing the brochure for and product managing the first Global Transit product (UUNET, 1996), I would like to clear up some history.

    Everything before that was a long line into MAE and trying to connect to as many as you could there. ISPs paid for the long line. We launched transit to take the long line closer to them (fewer hops).

    There is no difference between Transit, Paid Peering, buying from a CDN or buying hosting in an ISPs network. Each is simply a way of getting your content closer to your customers than you can get from free peering. Fewer hops.

    Everything else is simply features and marketing spin. The reason for choosing one of the other is now and has always been to reach the big user bases that exist on large business and consumer networks. In my day it was AOL everyone wanted to get to and they were on UUNET, so other ISPsv bought Global Transit. Now they buy Akamai because their servers are inside many ISPs giving the shortest hop route.

    Be sure of one thing – any legislation that seeks to prevent market forces delivering what customers want are doomed to fail. There are just too many clever engineers and lawyers around.

    Share
  28. [...] Net Neutrality.  I’ve written an article raising similar concerns as William Norton as did Richard Bennett, and we think this is a very legitimate question that must be clarified by the FCC.  This is [...]

    Share
  29. Here’s a nice summary of what I’ve been trying to say here: “We also have a cautionary conclusion: if one should be motivated (for whatever reason) to contemplate some regulatory rule to manage interconnection (which the debate over Net Neutrality is, in part, about), the design of such a rule will be both complex and informationally demanding. Any simplistic rules that try to define network neutrality as the elimination of discrimination will fail even to match today’s reality by a wide margin. There is a substantial level of economic discrimination today just in the variation in willing to peer, and the emergence of paid peering and partial transit only increase this space. Partial transit and paid peering may be seen as efficiency-enhancing responses to changing market conditions. While there may be opportunities for abuse by providers with excessive bargaining power, the complexity of what is in place today, and what seems to be working today, would argue that the best way to address any potential concern would be to focus on the sources of bargaining power and identify anti-competitive opportunism, rather than to impose ex ante restrictions on the range of bilateral contracts.” – Complexity of Internet Interconnections: Technology, Incentives and Implications for Policy, P. Faratin, D. Clark, P. Gilmore, S. Bauer, A. Berger and W. Lehr. http://people.csail.mit.edu/wlehr/Lehr-Papers_files/Clark%20Lehr%20Faratin%20Complexity%20Interconnection%20TPRC%202007.pdf

    Check it out.

    Share
  30. “Most credible sources – from David Clark to Patrick Gilmore to Bill Norton – regard paid peering as an emerging and valuable phenomenon.”

    Therein lies your problem. Bill Norton is not an expert on the subject of peering. He worked in Equinix’s marketing department; there’s no tangible evidence of him having negotiated or configured even ONE peering session. I think you’d get a lot further with your argument if you’d have consulted with an actual subject-matter expert on your research.

    Share
    1. What a pathetic comment. Bill is enough of an authority that he’s cited by David Clark – former chief architect of the Internet and FCC expert witness – and I shouldn’t rely on him because some blog reader who hides his last name has a grudge against him?

      The credibility arrow is not pointing in your direction, Dave.

      Share
      1. How about some facts, Richard, can you name ONE peering relationship Bill set up in his career history?

        Share
      2. So you wouldn’t take a hemorrhoid treatment from a doctor who didn’t suffer from the affliction, Dave?

        Share
  31. Also, keep in mind that Bennett is a lobbyist paid by the likes of Comcast etc. He’s telling us what they want you to hear.

    Share
    1. Richard Bennett is speaking for his organization, not Comcast, on this subject.

      Share
      1. Does Comcast provide the ITIF with any means of funding though?

        Share
      2. Actually, I’m speaking on behalf of the Internet here, as I assume you are.

        Share
  32. Patrick W. Gilmore Wednesday, December 9, 2009

    Richard,

    Thank you for noticing my paper!

    David Clark and I presented the paper at TPRC in 2007, and an abbreviated version was published in Communications & Strategies, a peer-reviewed journal. (Abbreviated because the journal had a page limit. A bit of a shame, this is a complex topic, and more space is probably better when discussing it.)

    David, Peyman, Steve, Bill, Arthur and I started meeting in 2006 to discuss the paper. I was explaining that there were these things, paid peering and partial transit, which had been around for ever but no one seemed to know about it. (These two are not the same, which you would know if you read the paper.) There seemed to be a stigma attached to them, and they were seldom discussed. But they were an important piece of the Internet economy puzzle.

    I said in the paper, and stand by this assertion, that both paid peering and partial transit had become more common in the last few years. (Remember, this was 2006, so I’m talking 2004 or earlier.) I hypothesized this was partially due to the rise of broadband networks. The reasons are not terribly complex, but more than I want to describe in a blog comment.

    However, the important thing was not the amount of increase of paid peering or partial transit. It was the fact these existed and no one seemed to know about them – back in 2006. Regulators were making decisions on a far too simple model of Internet transactions. Peyman, Dave, Steve, Bill, Arthur felt it was important that people knew and understood this before making decisions on policies such as Network Neutrality. So we strove to educate the policy makers and general public. Again, remember I am talking about our meetings in 2006, regarding a topic which was not brand-spanking-new at that time.

    To summarize: You have found is years old document, started the year before it was published, which tried to educate people on the importance of practices such as paid peering and partial transit. Or put another way, you have given everyone proof that what you are discussing is neither new nor startling.


    TTFN,
    patrick

    P.S. Please stop calling other people pathetic for believing Mr. Norton is not as much of an expert as others. Bill is not an expert because Dave Clark footnotes him. I was in the room when we discussed Bill’s papers, believe me on this. Also note I co-authored the paper with Dave and Bill thanks me (as well as Richard Steenbergen and others) for explaining things to Bill in those very same papers which were footnoted. So if we are going to “argue from authority”, I am the only one both Dave and Bill acknowledge. By your own logic, I trump Bill, and I am telling you that Richard, Vijay, Dan, etc. are all an order of magnitude more clued than Bill on these topics.

    Share
    1. I know most of your co-authors, Patrick, and have appeared with them on various panels and at TPRC. I don’t think that peering contracts are so complicated that one needs to be a genius to understand them; they span the full range of terms between pure peering and pure transit. The actual issue is is how their terms interact with the FCC’s proposed anti-discrimination rules, and how constraints on them will impede or support innovation. Peering coordinators have no particular expertise on these second and third order policy effects.

      Those of us who know how the process is going inside the FCC are concerned about the potential fallout; the agency is scarcely any more clued-in to peering and transit today than it was in 2007, and the histrionics of the IRC crowd hasn’t helped.

      Share
      1. It sounds like you’re concerned about the fallout not because you’re a concerned technologist, but because Comcast etc are funding your lobby. Why have you ignored the numerous requests from numerous people on this thread to shed some light on the ITIF’s funding sources?

        Share
      2. I’ve stayed away from posting on here (or on nanog) until now, but…

        This is not about peering contracts (which deal with who pays who for what), its about there being three very different types of peering/transist at a technical level (the following assume you are the ‘junior’ party in the transaction):

        1. Full global transit – I tell you (and your customers) about the ‘world’ and the ‘world’ about you (and your customers) . You can chose to send traffic via this link (or these links) and the world can choose to send traffic to you. This it what people tend to mean when the say transit.

        2. Peering – I tell you about my network and my customers and you tell me about you and yours. Again, individual networks can chose to use this information (or not)

        3. Partial Transit – You tell me about your (and your customers) network and I tell you about my network, my customers networks and a subset of the world (based on various predetermined factors such as regional location, ‘peers only’ or something else)

        Once you have established which of these is the technical nature of the relationship you need to do the financials (so option 2 where neither side pays is what most people call peering)

        Getting these separated (the concepts of a technical definition of what the service is) from the fundamental issue of the contractual relationship that facilitates the establishment of the service is key. I remember being involved in ‘paid peering’ discussions from 2003/4 when smaller broadband networks in the UK wanted to get ‘peering’ from the larger UK broadband networks (mostly to cover the costs of transit but also to help with performance issues) but never outside the small group of technical/commercial people involved in the discussions as it wasn’t something that needing ‘productising’.

        Since 2006 I’ve been involved in the sale of ‘partial transit’ as a product (and this gels with Patrick’s time line very well) and it was very much a service that you could only sell to a technically minded person.

        Today ‘paid peering’/’partial transit’ are (in Europe at least) very common products that are sold to/by various different networks operators to help cover the costs of building networks.

        James aka boggits

        Share
  33. Patrick W. Gilmore Thursday, December 10, 2009

    “So you wouldn’t take a hemorrhoid treatment from a doctor who didn’t suffer from the affliction, Dave?”

    Of course a doctor doesn’t need to have a disease to treat it. But your metaphor is not on-point, in my opinion. Allow me to illustrate.

    Assume I go to a proctologist’s office and there is an intern working there. I tell the intern my gluteus maximus hurts due to hemorrhoids. If the intern tells me I am confused and stupid because hemorrhoids only happen on your rear end, then argues with attending when the attending tells him “those are the same thing”, no, I would not take treatment from him.

    Of course, it is entirely possible that intern knew exactly how to treat hemorrhoids. But the fact that 1) he was confused about The Most Basic Facts regarding his professed specialty, 2) argues with those who actually are experts in the field, and 3) is clearly unprofessional, all prove to any sane person that the intern should not be allowed near anyone’s ass.


    TTFN,
    patrick

    Share
    1. Patrick, you’re forgetting the fact that Richard is being PAID by big telco/cable to write this nonsense and argue it in Washington. Following your analogy, the intern would be getting kickbacks from pharma as well. :)

      I think one thing we can agree on is that Om really dropped the ball on this one, passing off this bumbling idiot as an “expert”.

      Share
    2. I find it interesting that you’re arguing against your own conclusions in the cited paper, Patrick. Remember: ““We also have a cautionary conclusion: if one should be motivated (for whatever reason) to contemplate some regulatory rule to manage interconnection (which the debate over Net Neutrality is, in part, about), the design of such a rule will be both complex and informationally demanding. Any simplistic rules that try to define network neutrality as the elimination of discrimination will fail even to match today’s reality by a wide margin. There is a substantial level of economic discrimination today just in the variation in willing to peer, and the emergence of paid peering and partial transit only increase this space. Partial transit and paid peering may be seen as efficiency-enhancing responses to changing market conditions. While there may be opportunities for abuse by providers with excessive bargaining power, the complexity of what is in place today, and what seems to be working today, would argue that the best way to address any potential concern would be to focus on the sources of bargaining power and identify anti-competitive opportunism, rather than to impose ex ante restrictions on the range of bilateral contracts.” – Complexity of Internet Interconnections: Technology, Incentives and Implications for Policy, P. Faratin, D. Clark, P. Gilmore, S. Bauer, A. Berger and W. Lehr. http://people.csail.mit.edu/wlehr/Lehr-Papers_files/Clark Lehr Faratin Complexity Interconnection TPRC 2007.pdf

      The paper cites Bill Norton 6 or 7 times.

      Do you now think you were wrong, or are you upset that I didn’t cite you to begin with? I wish I had, but I only had 600 words to work with; maybe next time.

      Share
      1. Wow, you’re dense aren’t you. Don’t you get it, the people you’re talking to are the ones who have been ghost writing Bill Norton’s white papers for the last 10 years. Do you really think Bill has any insights about peering? He knows next to nothing about the subject. He writes some random crap that isn’t even close to right, shows it to all the peering people that you’re arguing with here while playing the “nice confused guy” routine, and they feel sorry for him and write correct versions for him. He spent his years at Equinix in marketing, churning out crappy white papers just trying not to get fired. Why do you think he left, because he had another gig somewhere? It was because they were on to his game, and were about to can him. His “papers” are the laughing stock of the industry, and the people citing him anywhere were just trying to be nice.

        Share
      2. In response to Dr Depeering,

        I can see why you’re posting this anonymously. This is some of the most vile and libelous comment on a public forum I’ve ever seen.

        Share
      3. I had a lot more respect for peering coordinators before publishing this blog post.

        Share
  34. Patrick W. Gilmore Thursday, December 10, 2009

    “I find it interesting that you’re arguing against your own conclusions in the cited paper, Patrick.” I do not believe I am arguing against my own conclusions. As the author of the paper, I am confident my interpretation of my own words is superior to yours.

    Even assuming your interpretation of the paper (private peering is new when the paper was published), you are years late to the party. This is nearly 2010, that paper was published in 2007. Are you still claiming that something that paper noted years ago is now a new phenomenon? Do you have any idea how long 2-3 years is on the Internet? When Peyman first brought us all together to discuss the paper, Comcast did not have a cohesive background, and Google had not bought YouTube.

    The thread of these comments has wound around to the point where I cannot even tell where you stand. Wherever it is though, I am sure it is not any place I recognize.

    You accuse me of arguing against a paper I published years ago, yet you cannot see it contradicts you.

    When Peyman & I cite Bill, you claim this makes Bill an authority. Yet when Bill cites me or Richard S., we have no authority.

    You talk to a colo provider, they say there are 300 x-conns / month, you assume a bunch are paid peering. When people who actually ordered and used those x-conns tell you very few are for paid peering, you disagree.

    Etc., etc.

    You deny reality. You resist logic. Your understanding of how Internet networks interconnect is woefully inadequate, and worse, factually incorrect. Both eyeball networks and content networks, NN advocates and critics, large companies and small, they all tell you you are wrong. But instead of listening to everyone, you staunchly resist everyone. I’m not sure which side you are taking, but whoever it is, they are not taking your side.

    “It is impossible to defeat an ignorant man in argument.” -William G. McAdoo, lawyer and politician (1863-1941)

    Good advice.


    TTFN,
    patrick

    Share
    1. Given that the Internet is 40 years old, I don’t think a paper delivered in two years ago is completely obsolete. We’re currently dealing with the issue that your conclusion highlights, the scope and extent of the anti-discrimination rule.

      I said in my 600 word blog post: “Paid peering illustrates how hard it is to write an anti-discrimination rule for the Internet that doesn’t have harmful side effects for all but the largest content networks. Paid peering is a better level of access to an ISP’s customers for a fee, but the fee is less than the price of generic access to the ISP via a transit network. The practice of paid peering also reduces the load on the Internet core, so what’s not to like? Paid peering agreements should be offered for sale on a non-discriminatory basis, but they certainly shouldn’t be banned.”

      and you said in your conclusion:

      “We also have a cautionary conclusion: if one should be motivated (for whatever reason) to contemplate some regulatory rule to manage interconnection (which the debate over Net Neutrality is, in part, about), the design of such a rule will be both complex and informationally demanding. Any simplistic rules that try to define network neutrality as the elimination of discrimination will fail even to match today’s reality by a wide margin. There is a substantial level of economic discrimination today just in the variation in willing to peer, and the emergence of paid peering and partial transit only increase this space. Partial transit and paid peering may be seen as efficiency-enhancing responses to changing market conditions.”

      Is there a contradiction?

      The question isn’t whether paid peering and partial transit are new in the peering space; they’re “new” in the policy space insofar as the regulators are just now learning about them (I exempt Doug Sicker from this group of regulators, as he obviously understands these things quite well.) So I’d encourage you to get off the tangent that Google’s man started and join the real discussion.

      Share
      1. “Paid peering is a better level of access to an ISP’s customers”

        Could you please site some examples?

        “the fee is less than the price of generic access to the ISP via a transit network”

        Examples and sources here too, please.

        “The practice of paid peering also reduces the load on the Internet core, so what’s not to like? Paid peering agreements should be offered for sale on a non-discriminatory basis, but they certainly shouldn’t be banned.”

        It has hard to argue this point when your previous two statements are missing citations.

        Please don’t quote Norton either, we’ve already proven he’s not an expert on this subject.

        Share
    2. Patrick,

      You keep ignoring what the actual debate, the ORIGINAL debate is about. The question that Norton (followed up by me and Bennett) raised was not to get into a semantics argument about what’s new/old, what is/isn’t paid peering, and who is and isn’t an expert. Never did any of us imagine we’d get into a flame war like this and some of the attacks here have gotten beyond ludicrous.
      The key question is whether it is wise for the FCC Net Neutrality regulation to regulate Paid Peering. In the paper that you coauthored two years ago, you warned about the dangers of Net Neutrality trying to regulate and prohibit interconnects and Paid Peering in the name of ending “discrimination”. Now that we are actually facing these concerns about interconnect regulation from the FCC NPRM (and trust me when I say we have more inside sources than you), Bill Norton has raised the flag.
      Now you and a bunch of folks come here and say Bennett and Norton are wrong and you attack them claiming that they are not experts (even though you cited Bill Norton at least 6 times as a peering expert two years ago). But why go down this road of character assassination in the first place? If you want to disagree with our interpretation, that’s fine. But why be so defensive that someone would suggest that we get clarification from the FCC and debate this thing in public which is exactly what the FCC has asked us to do?
      What’s the worst thing that can happen if we debate this? The FCC will come out and say Norton’s interpretation was wrong and there was no intention of regulating/prohibiting certain types of interconnects? Now what if Norton was right, and we all sat around and said nothing as the FCC starts banning Paid Peering? That’s essentially what you and JZP are saying, that we should just shut up about it because we don’t know anything.
      Really, I don’t get the source of this animosity. If all of us agree that an FCC ban on Paid Peering would be bad, then why can’t we come together on this one simple point? We can agree to disagree on interpretations, but let’s ask for clarification and wait and see.

      Share
      1. And on the FCC point, one has to look no further than this Tuesday’s workshop on the Net Neutrality regulations to see that peering agreements were discussed extensively: http://www.openinternet.gov/workshops/ws_tech_advisory_process.html

        So yes, the FCC is considering whether and in what form the net neutrality regulations should extend into the peering and transit space; one reason they hired Doug Sicker is for his expertise in this area, as he’s a former chief architect for Level 3. So no, it’s not at all unreasonable to consider this question, whether one happens to be a peering coordinator or not.

        Share
  35. Patrick W. Gilmore Friday, December 11, 2009

    I’ve given up trying to educate Richard, it is not possible. But I figured the others reading this may need a little clarification on Richard’s last comment.

    I should state first that I agree the tone of “Dr. Peering”‘s comment was less that polite. I do not know who wrote it, and I hope everyone will note that not one of my posts had such a tone.

    Unfortunately, whatever the tone of the comment, the substance is a genuine reflection of how many professionals in the field feel about Bill Norton’s white papers. Having read his original drafts before publication, I can tell you they are riddled with errors, some due to lack of experience, some from vast oversimplification.

    Bill tries hard to explain complex ideas to people, a laudable goal, and many of us tried to help him in the past. But despite his prolific white-paper publication record, Bill Norton is not an expert in. The fact Richard believes Bill is an expert only shows, yet again, that Richard is not.

    Also, most of the people posting here are not “peering coordinators”. That Richard harps on this again & again, despite being told otherwise repeatedly, proves not only his ignorance of the issues and people involved, but his inability to either research or even read. There is one person who made a comment here and is a peering coordinator, but she knows a metric ass ton about policy, putting to lie Richard’s comments even when he is talking about peering coordinators.


    TTFN,
    patrick

    Share
    1. The purpose of the blog post is not to confirm or deny Bill Norton’s or anyone elses position as the world’s leading expert on peering and transit, it was to highlight an issue with the pending net neutrality anti-discrimination rule that warrants discussion and review. I’d suggest that you’re not covering yourself with honor by harping on the credentials of a source, especially one that you yourself have used quite extensively, Patrick. When you’ve dug yourself a hole like this, it’s best to stop digging.

      Share
      1. Richard, the problem as I see it is that your entire article relies on “expertise” from Bill Norton. You did no original research, you just paraphrased the DrPeering blog post. Considering your research is fundamentally flawed, how can you argue your position is valid?

        Also, aren’t you being paid by big cable/telco?

        Share
  36. [...] Internet has adapted to the video challenge by shifting the majority of Internet traffic from a dozen Tier-1 network providers to the edges of [...]

    Share

Comments have been disabled for this post