Peering is essentially an arrangement between two bandwidth providers where they send and receive traffic from each other for free. The logic is that the data sent from one network to another is reciprocated. Verizon runs one of the largest last mile networks and owns the descendants of MCI. Cogent is one of the largest bandwidth providers, and its network is spread across the globe in hundreds of cities.

Cogent and Verizon peer to each other at about ten locations and they exchange traffic through several ports. These ports typically send and receive data at speeds of around 10 gigabit per second. When the ports start to fill up (usually at 50 percent of their capacity), the internet companies add more ports. In this case, through, Verizon is allowing the ports that connect to Cogent to get crammed. ”They are allowing the peer connections to degrade,” said Dave Schaffer, chief executive officer of Cogent said in an interview. “Today some of the ports are at 100 percent capacity.”

“Think of it as the on-ramp to the freeway being log-jammed,” Shaffer said. And that means your Netflix content, especially content sent by Netflix’s content delivery network, slows down, and you get pixelated pictures and buffering.

While not naming Netflix directly, Verizon has indicated to Cogent that the reason behind its actions is that Cogent is moving traffic for a large video provider. Schaffer confirmed the Netflix is one of their largest customers. “Over the past year Netflix has become a big partner for us. This is a business model problem, not an engineering problem,” Schaffer said.

Our sources tell us that Netflix recently bought 2 Terabits of bandwidth capacity in part to get around such cramming that was happening in places where it sends traffic directly to certain internet service providers.

When we called Verizon about this story asking if Verizon was having a problem with Cogent over peering issues associated with Netflix, Verizon spokesman Bill Kula said he’d get back to us. A few minutes later he sent the following reply that didn’t answer our question:

Verizon operates one of America’s lowest-latency, highest capacity networks. The various classes of Internet speeds we offer are among the fastest in the nation. Time and again, customers rate us best in class in various reports and surveys. Our customers enjoy a consistently superior Internet experience because our networks can adapt and grow with their use

Netflix has been growing like crazy and it now accounts for a whole lot of Internet traffic — almost one out of every 3 bits (32.3 percent) sent downstream to users in North America is Netflix traffic according to Sandvine, a company that makes traffic monitoring gear for ISPs. That’s a lot of congested ports.

Netflix’s growing popularity has made it a target of ISP (internet service providers) vitriol and anger, especially those who offer competitive services. Verizon, for instance owns 50 percent of Redbox, a video-over-the-Internet service that is competitive with Netflix. Time Warner Cable and Comcast are other large providers that has allowed degradation of the online video experience on its networks — after all the logic is that as people start to have a bad Netflix experience, they start to look for alternatives — perhaps the ISP’s own pay TV offering.

This isn’t the first application last mile network operators have tried to degrade — last year the wrath of the Baby Bells and cable companies fell on Megaupload, a file sharing company started by Kim Dotcom, Schaffer said. That too was one of the big bandwidth-hungry services popular with the end customers of the ISPs — actual consumers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• OTT technologies and strategies for broadcasters
• What the shift to the cloud means for the future EPG
• Who and what to watch in the new era of the living room

As part of its investigation, the regulator is also probing three other unnamed companies. The perception is that ISPs in France are either under-investing in infrastructure or violating the spirit of network neutrality, the idea that ISPs should not discriminate on traffic traveling over their pipes. Yet, in France, it seems that at least some in the government are willing to make Google pay for the ability to guarantee that ISP customers can receive its bits, turning the internet into Gulliver in the land of the Lilliputians, with ISPs and governments tying it down. The question is, will what happens in France happen elsewhere?

The ARCEP investigation and user complaints

Early this year, communications regulator ARCEP will rule on an investigation it opened on November 22 following complaints that video streaming services including YouTube are often too slow to watch. Now three French senators are also calling on the country’s digital economy minister to take action.

ARCEP stepped up when a survey of over 16,000 ISP customers by French consumer group UFC Que Choisir found 83 percent of Free customers, 47 percent of Orange customers and 46 percent of SFR customers were unable to use YouTube properly.

Since the investigation began, many users have reportedly noted an improvement in connection quality, but connections remain patchy.

And this issue is not limited to Google. “The quality of connection is inadequate in almost all operators,” said UFC Que Choisir’s survey, which found that France’s native Dailymotion, ironically, is slowest to access through France Telecom’s own Orange ISP, which owns almost half of the YouTube rival. And 25 percent of consumers reported slow-downs while watching broadcaster TF1′s MyTF1 catchup service.

Is it under-investment or a desire to make content companies pay?

UFC Que Choisir says these symptoms may be caused by under-investment in infrastructure as well as commercial tensions between ISPs, which deliver web services to the end user but which don’t pay for the infrastructure. It has called on the government to define quality-of-access rules, in part by allowing the competition regulator ARCEP to build a quality-of-service observatory. Similarly, in the U.S., the Federal Communications Commission is also trying to figure out how to measure the quality of a broadband connection beyond just looking at speeds.

Inter-company tensions do appear to be at play. A case before another public agency — France’s competition regulator, the Autorité De La Concurrence — in September illustrates how ISPs eager for revenue from web content companies can hold the user experience hostage.

Cogent, which handles YouTube’s peering interconnections, had complained to the competition authority that Orange had refused its connections, wanting more money to add ports to connect Cogent traffic to its networks. Much of the interconnections between large ISPs, CDNs and web content companies are negotiated by private deals, so it’s rare to see the government get involved, or even to hear much about them publicly. In the U.S., when Level 3 and Comcast became embroiled in a public peering fight after Level 3 started sending Netflix traffic over its connections with Comcast, the FCC refused to get involved, and both parties settled the disagreement.

But in this case, the competition regulator said one ISP was within its rights to charge more money from services hoping to reach its subscribers. This so-called double-sided business model has been sought by ISPs who argue that companies such as Google are freeloaders making huge profits off the pipes of owned by the ISPs. In contract, Google and other content companies argue that their services are the reason customers upgrade to higher speeds and continue paying ISPs money.

And in France the ISP argument has gained at least one supporter in the Autorité De La Concurrence. The authority says France Telecom is offering interconnection prices significantly below market value, and it has accepted the telco’s undertakings to ensure transparency.

French policymakers generally are generally in the mood to extract more money from Google. They have already set such wheels in motion around taxation and copyright fees. Now infrastructure could be the next arena. But its unclear how far France will go.

It’s one thing to try to ensure that consumers have an acceptable connection to support online video, through implementing some kind of standards. It’s another to get between participants in peering disputes, and to possibly start setting rates.

The U.S. fight is bigger than France’s

But such fights are becoming more common as the stakes over the internet and web video get higher. ISPs are worried about the cost of delivering video traffic over their networks, while also losing out on the ability to charge users for pay TV packages that significantly boost their revenue. Meanwhile, consumers are demanding more video online because they can choose what to watch, when they want to watch it, on any device.

The French competition authority’s earlier investigation referred only to two individual companies at loggerheads in a specific peering fight. But the new inquiry by the communications regulator is much wider, looking at industry-wide practices around infrastructure and interconnection. And it comes following consumer outcry — identified in UFC Que Choisir’s survey — to which politicians may be keen to respond.

The outcome will be important because Google is already facing having to pay to re-use news content in some parts of Europe through both commercial agreements and possible new legislation that suggest excerpting is chargeable. That could set new precedents for the online content economy.

The French broadband outcome will be important because it will set up a precedent for other telecom regulators who are struggling with similar issues. Creating standards to ensure that customers have a quality online video experience is no simple task and may well require investment by ISPs, especially those on older technologies or oversubscribed networks.

As for the peering issues, if the government decides to step into the fray there, it could be setting the internet up for regulations that put governments in the role of determining who can connect to whom and how much they can charge. The OECD recently made a convincing argument that such regulations and government involvement would hurt the web and the economic development of companies dependent on the web. So what France does here might have influence far beyond its borders.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Connected Consumer 2011: What Not to Expect
• How consumer media will change in 2013

Update: Research firm TeleGeography emailed us that three international submarine cables in the Mediterranean Sea were damaged today, causing disruptions to internet and phone traffic in Egypt, Saudi Arabia, India and all of the Gulf states. TeleGeography pinpoints the faults  between Tunisia and Italy, and claims the damaged cables are the FLAG Europe-Asia cable, operated by Reliance Globalcom, and two consortium cables, SeaMeWe-3 and SeaMeWe-4 owned jointly by several telecommunications companies. From the TeleGeography statement:

The current series of faults is reminiscent of the submarine cable faults that occurred in January 2008. Today’s events have the potential to create worse disruptions: while the January 2008 accidents broke two of the three cables connecting Europe with Asia via the Middle East, Friday’s cable failures have caused faults on all three. France Telecom projects that service on all cables will be restored by December 31. Until then, many carriers in the Middle East and South Asia will need to route their European traffic around the globe, through South East Asia and across the Pacific and Atlantic oceans.

It’s unknown if the Malta cable problems are related to these cuts, perhaps from a weather or seismic event. However in the online world the cuts are certainly related in how they will make it that much slower or impossible for users to connect around the world. (Malta cable cut report published earlier follows.)

In a reminder of both the frailty and the flexibility of the web, the Times of Malta is reporting that last night, a submarine cable delivering traffic to subscribers of GO’s broadband service experienced a “fault.” Thousands of Maltese lost their web connections.  Combine Malta’s experience with the earlier epidemic of a few undersea cables getting cut over a period of days, and a fight by Sprint and Cogent in the U.S. over peering agreements that cut off the web for some users, and it becomes clear that we should consider the web not only as physical infrastructure, but also held together by political and economic agreements.

It’s like an information superhighway, but also a like series of treaties that allow trade to various points of the globe. In Malta’s case, an agreement with Vodafone to share its cable kept the physical infrastructure from staying out. But as the Sprint/Cogent peering fight proved, when those agreements fail, the web is vulnerable in a way roads are not.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of mobile: a segment analysis by GigaOM Pro
• Where new opportunity lies in the mobile operating system space
• Consumer privacy in the mobile advertising era

Sprint initiated a temporary reconnection to the Cogent network on Sunday, November 2nd so that customers would have temporary access while longer-term alternate and permanent access options are explored. We emphasize that this reconnection is temporary only, as the core issues in this dispute have not changed. Cogent was notified in advance of the November 2nd reconnection; therefore, any access disruptions occurring during this temporary period are the sole result of a negative reaction instigated by Cogent against the customers of both parties.

Earlier this week when I spoke to Dave Schaffer, CEO of Cogent Communications, he pointed out that Sprint was a tiny fraction of the traffic coming to Cogent network, and the dispute was affecting customers who are single-homed to only the Sprint network, especially those trying to access the Internet from Sprint’s mobile devices.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• CES 2012: a recap and analysis
• 12 tech leaders’ resolutions for 2012

What I’m talking about is wholesale Internet bandwidth that is sold to Internet services providers (ISPs) and content companies like Yahoo and Google. This is called IP Transit and it is sold at a rate of “per megabit per second per month” and often requires a monthly bandwidth commitment. Cogent Communications, Level 3 Communications, Tata Communications, Global Crossing and AT&T are some of the more well-known IP Transit providers.

Today research firm Telegeography came out with a report that shows the price of wholesale Internet access (IP transit), while varied around the globe, are still in decline. Here are some facts.

• GigE port prices in major U.S. cities fell 30-40 percent between Q2 2007 and Q2 2008. Median monthly IP transit prices for 1,000 Mbps Gigabit Ethernet (GigE) ports in major U.S. and European cities ranged from $10-$14 per Mbps in Q2 2008.
• GigE port prices in Latin American cities declined a more modest 15-20 percent for the same period. Median GigE port prices range from $73 per month in Buenos Aires to $86 per month in Santiago.
• Prices for GigE ports in major Asian cities in Q2 2008 ranged from $30 per Mbps month in Seoul to $45 per Mbps per month in Tokyo, higher than the U.S. or Europe. The price declines were around 30 percent.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Confused about the wireless markets? Here’s a breakdown
• GigaOM Euro 20: the European startups to watch
• Survey: How apps can solve photo management

The bickering could be because one of the two parties feel they need to be paid for the traffic they are sending. I am looking into this and have emails out to all my sources in the bandwidth business, and will update accordingly. (If you have any information, drop me a line.)

Cogent had previously gotten into a slugfest with Level 3 Communications. Cogent’s stance towards Telia is hypocritical. Here is a press release they issued back in 2005 when complaining about Level3. As a reader points out, Cogent has been involved in such spats with other carriers as well. OpenTransit (France Telecom), Teleglobe, ATDN are some of those who have had Cogent issues.

This is a letter Telia sent to its customers:

Dear Customer,

We would like to give you the following information:

Cogent has decided not to exchange traffic directly with TeliaSonera’s AS 1299 or indirectly with AS 1299 through a third-party provider. As a result, Cogent has partitioned the Internet and disrupted the flow of traffic between Cogent and TeliaSonera customers.

While this has a negative impact on some users of the Internet, this effect is the result of Cogent’s decision and is unfortunately beyond TeliaSonera’s control. Until Cogent rectifies this situation, TeliaSonera customers experiencing any difficulty reaching Cogent’s network can continue to purchase IP Transit from TeliaSonera along with another Tier 1 provider. This will fix the immediate problem and ensure optimal connectivity going forward.

We sincerely apologise for the inconvenience caused…

If you have further concerns, please address your commercial contact at TeliaSonera

Updated with inputs from our readers. Thanks guys

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Survey: How apps can solve photo management
• Social networks will displace business processes, not socialize them
• Sector RoadMap: Social customer service in 2013