Yesterday Mathew Prince, the CEO of Cloudflare posted some awesome data on his company’s blog showing that the U.S. has a higher cost for bandwidth than Europe. He’s not the only one with such data, but he’s one of the first to discuss it so openly and to dive into how the costs of peering versus buying transit affects those costs.
First off, transit is where a company buys wholesale bandwidth on a per gigabyte basis from providers that can range from Level 3 and Tata to companies like Comcast or AT&T. Peering on the other hand, is a direct link to another providers’ network that can be paid or set up as a free exchange of traffic.
I’d like to add to Prince’s insights with complementary data that came out in July on the costs associated with buying cross connects in data centers. As Prince (pictured above) explains in his post, these cross-connects are a part of the overall cost associated with setting up direct peering agreements. From his post:
Because of the high rate of peering and the low transit costs, Europe is the least expensive region in the world for bandwidth.
The higher rate of peering is due in part to the organization of the region’s “peering exchanges”. A peering exchange is a service where networks can pay a fee to join, and then easily exchange traffic between each other without having to run individual cables between each others’ routers. Networks connect to a peering exchange, run a single cable, and then can connect to many other networks. Since using a port on a router has a cost (routers cost money, have a finite number of ports, and a port used for one network cannot be used for another), and since data centers typically charge a monthly fee for running a cable between two different customers (known as a “cross connect”), connecting to one service, using one port and one cable, and then being able to connect to many networks can be very cost effective.
So in places where cross-connect costs are high, it can make less sense to peer until you’re a much bigger player because instead of paying fees associated with direct peering, buying transit on a per gigabit basis can be more economical. Transit prices continue to drop! And in the U.S. cross-connect pricing is high, as this chart from TeleGeography shows.
Jon Hjembo, a TeleGeography analyst, explains the differences in U.S. versus European cost-connect pricing as a matter of business models between exchanges (Prince does too). Hjembo notes that in Europe, third-party internet exchanges are much more common where different providers and exchanges all co-locate in data centers. These hubs make it easy for many providers to directly connect their networks at a low cost.
“Due to the established market dynamic there, connectivity would never be a money-maker for European colo [co-location] providers even if they wanted it to be,” Hjembo said via email. “In the U.S., there’s a history of prominent colo providers offering their own IXs [internet exchanges]in-house. Operators can charge premiums for access to more network-dense ecosystems.”
So business models like data center provider Equinix’s mean that connectivity costs more because as a single provider of a neutral meeting point for carriers those connectivity costs are where it makes some of its revenue. However, that model may be shifting as companies like Netflix and Digital Realty push to bring the European model to this country. Hjembo also noted that in the data above he couldn’t get information from every single market player, but if it were universal he’s “certain the disparity would be even higher.”
What about paid peering?
So we have higher interconnection costs for smaller providers in the U.S. that can lead to higher set up-costs for peering, which in turn raises U.S. bandwidth prices. We also have a regulatory vacuum and an uncompetitive last-mile broadband market that’s further distorting the market for bandwidth costs. Together these two things could drive up prices, and even if the cross-connect pricing drops, the last-mile monopoly might end up pushing prices up faster than they might be lowered at the data center.
As Prince notes in his piece, peering in the U.S. is becoming a hot topic thanks to Netflix fighting with the major U.S. ISPs for the ability to deliver its content to subscribers at the top four ISPs without having to pay for direct interconnection. There are two types of peering– the type where two networks interconnect because it’s beneficial to them and which is how 99 percent of the peering agreements happen or paying the provider a fee to peer with them based on how much traffic is sent over the link. Netflix now pays to send traffic to AT&T, Comcast, Time Warner Cable and Verizon.
We’ve covered why this is a problem, as does Prince, but Dave Burstein, a respected writer covering telecommunications also does a good job pointing out that while the current prices ISPs are charging Netflix for interconnection rates are low, they represent a shift in the network neutrality fight. The issue with network neutrality is that we need it because the last-mile broadband market is not competitive. If a consumer’s ISPs starts blocking content or charging content companies more to deliver their bits, the consumer can’t vote with their feet.
The FCC has decided that network neutrality only affects the last mile connection between the consumer and the ISP’s network. Meanwhile, peering fees are charged where a content provider wants to get onto the ISP’s network, putting it outside the net neutrality fight. But the lack of competition is still a problem, and regulators need to be aware that ISPs are seeking to exploit that vacuum for their own gain, as Burstein reports.
He starts by saying he has a “very senior person” at one of the largest U.S. ISPs confirming that his company intends to collect on these peering fees. It’s a concept called sender pays and people in the telcos have been arguing for that for decades as a way to keep their revenues in line with their 20th century infrastructure and business models. Burstein writes:
Sender pays is a mistake because the carriers have enormous market power and are likely to use it. There’s no way to reach 21 million homes without going through Comcast. That’s called a “terminating monopoly.” Broadband customers typically won’t change ISPs for 5 years or more so Comcast’s control will continue. A video company can’t make it if blocked from a quarter of the customers in the U.S., especially if the other ISPs do the same. With no choice, video providers will have to pay even exorbitant rates. Some of that inevitably will be passed on to consumers.
Brian Fung over at the Washington Post has also written a clear outline of how peering and transit work and why it’s important (I am especially glad because lawmakers and their aides read the Post and the article could help them understand why it’s such an essential topic.)
For those who don’t care about the cost of bandwidth, or think all of this is an esoteric argument that will ultimately have little effect on the internet, let me disabuse you of that notion. The transactions that Prince is writing about, either buying transit or negotiating interconnection or peering agreements is at the heart of bandwidth creation. While the pipes in the ground have a set limit on the capacity they can transmit and cost money to upgrade, the electronics to fill those pipes are pretty cheap on a relative basis. Thus, once you have high capacity fiber in the ground connecting places, you can transmit at that capacity with fairly predictable operational costs.
One reason the internet is such a successful platform for innovation is that it provides a cheap distribution model for everything from computing to cat videos. Things that raise the cost per bit — be it monopolies charging rents to get data to the consumer or businesses that want to take a cut from linking two networks together — raise the cost of that distribution and can impede it. And that’s going to change the internet as we know it — likely not for the better.