22 Comments

Summary:

The internet is made of thousands of networks, and a complex web of economic considerations has developed to support the free flow of information. How bandwidth is “manufactured” and then allocated is far more complex than how a packet gets from here to there.

amsIX
photo: Om Malik

Most people know certain things about the Internet. They know that cables run under the sea, that wires come into your homes, and that modems carry the digital signals to your devices.

But they’ve probably never heard of Internet Exchange Points, and that’s where the magic of the Internet really happens.

Internet Exchange Points (aka IXPs) are the manufacturing floor of the Internet — that is where bandwidth is created and deployed. And bandwidth is just like water and oil and other economic goods: If your country has a lot of it, prices fall; if it doesn’t have a surplus, prices go up. And that has a big impact on the web companies that buy bandwidth.

The Netherlands, for example, is a large net-exporter of Internet bandwidth, using only half of what it produces domestically. That means that large companies like Disney, Google and Netflix can buy ports there at rates that are significantly lower than in some other places and that have dropped by 50 percent in the last year. But where there isn’t an Internet Exchange Point and competition can’t flourish, prices remain high. That’s what you see in places like Mexico. More on that below.

Though Internet Exchange Points are a key building block of the Internet economy, you’d never guess it from where they’re located. They don’t remotely resemble other temples of commerce.  You don’t see people standing in them yelling at each other, like you would at a stock exchange. Instead, they can be located  inside a beat-up building near a railroad track. Sometimes there’s not a single human being in the vicinity.

A report I covered earlier from the OECD lays out in awesomely clear detail why Internet exchange points are so essential for every geography that values the Internet. They not only facilitate the creation of bandwidth, they lower the cost of transit/bandwidth for businesses and consumers and help create redundant networks and limit or diminish the power of monopoly telecommunications providers.

The report does a great job of explaining exactly how the internet economy functions. Not the creation of web sites and services (although the report does delve into the creation rationale for “the cloud”) but bandwidth is created and then used. It explains why certain places have higher Internet transit costs and why it makes sense for different Internet Service Providers to essentially exchange traffic. It offers a compelling look at how important competition among network providers is to keeping the internet cheap.

From the report:

Internet exchange points (IXPs) are the source of nearly all Internet bandwidth. A country that lacks IXPs must import Internet bandwidth from other countries that do possess them. Like factories and farms, they are a primary means of producing a commodity that‘s potentially quite expensive to import.

As a general rule, the cost of telecommunication services is the product of the speed of the service multiplied by the distance covered (i.e. speed x distance = cost). The further afield you go for your bandwidth, the more expensive, and slower, it will be. Thus it‘s always preferable to use a local IXP, to one that‘s further away.

As a result, IXPs proliferate in areas where Internet service providers, users, and policy makers are well-informed on matters of telecommunication economics. A corollary is that a region which has many functioning IXPs and produces more bandwidth than it consumes can export bandwidth to other regions at a profit.

This formula of cost as a matter of speed and distance is important to understand why peering relationships are so important. Not only does a provider not have to build a network that covers every user in the world, but it can exchange traffic with other providers that also don’t want to build out networks. And because most of the peering is done via handshake agreements with no money changing hands, the operational costs of sending traffic are pushed as low as the network owner can get them.

It doesn’t mean the providers don’t pay any costs, but that the multiple providers essentially are building quid pro quo relationships so everyone doesn’t need to build out a network to cover every single person who wants to be on the internet. And while some traffic may be slightly imbalanced for some providers, it keeps prices low for end users, which means they consume more bandwidth, which is good for the whole ecosystem. Here’s what those peering relationships look like in chart form:

Sometimes this utopian vision doesn’t take place.

Mexico offers an example of what happens when there’s little competition, either from another bandwidth manufacturing entity or from the existence of an IXP that can help offload traffic for many providers. As the chart below shows, you get disproportionately high prices for regional transit, which can then be passed on to consumers. For example, in parts of Africa, consumers and businesses paid higher transit prices, but the construction of more submarine cables and an Internet Exchange point has helped lower costs.

From the report:

Mexico continues to lag, being by far the largest nation in the world, and the only OECD member country, to continue without any domestic Internet exchange capacity. Mexico‘s traffic continues to be exchanged largely on the East Coast of the United States, and to a lesser degree in the exchanges of its Latin American neighbours and European trading partners.

This is a consequence of the near-universal dominance of Mexico‘s incumbent, Telmex. This situation may be on the brink of reform, as COFETEL, the Mexican regulator, has opened access to competitive long-haul circuits, has licensed a second national carrier, and is investigating the establishment of an IXP. The lack of domestic traffic exchange has had a dramatically visible effect on Mexican transit pricing, relative to other economies of similar size and development:

Outside of regions that don’t have an IXP, there are other threats to this model of how the Internet currently works and regulates itself, namely regulations at the global or national level that might clog the free-market nature of these exchanges. Interestingly this report makes a good case that even actions taken by large ISPs or backbone providers to restrict who they peer with doesn’t cause very much economic harm to the Internet as a whole. It shows how many of those choosy providers are bypassed by those they refuse to peer with and those rejected peers find another network.

Given how important broadband and Internet access is becoming to our lives and economies, the report provided a lot of good data and good news. Check it out.

You’re subscribed! If you like, you can update your settings

  1. Stacey,
    I generally enjoy reading your posts, but this one illustrates certain gaps in understanding of how the internet works and grows. Did you proof this with any architect at a sizable network that actually make the internet happen?

    1. gt, do you want to be more specific in your objections to her write-up? What exactly in her report are you objecting to?

    2. Stacey Higginbotham GT Friday, November 2, 2012

      Glad you like my writing, and while I don’t have network architects proof my articles, I do talk to them on a consistent basis. This story relies on that base of knowledge and the report. That being said condensing a 92-page report and an overview of internet economics into a 1,000-word article may leave gaps or perceived gaps. I’m happy to address those if I know what they are.

      1. The commenter elfonblog (below) does a pretty good job explaining the issue, albeit a bit harshly.

      2. Thank you, Jared. I apologize for my harshness, Stacey. I wrote while in the grip of my passion for this subject. I am a veteran ISP employee (Illuminati Online, Onramp Access and teleNetwork), and an experimental wifi hobbyist.

      3. Hi Stacey. My comments were based on two points.

        One being the concept of “manufacturing” bandwidth, as was already discussed below. The hardware underlying the internet infrastructure costs everyone roughly the same. The cost to trench fiber is roughly the same everywhere. If a company or country is charging more per unit of bandwidth, it’s market related, not cost related.

        The other is the over-reliance on the Internet Exchange Points that you made. The *vast* majority of inter-connectivity in the world does not take place across IXPs. To explain why a Netherlands for example has as much traffic flowing through it’s IXP despite having 1/5 the population of Germany or 1/4 the population of England, all you need to do is follow the fiber. Many companies build there because it’s an important intersection point of subsea and terrestrial fiber, so most of the traffic there is being exchanged between companies that aren’t based there. The utilization per subscriber in the Netherlands is actually inline when compared to other OECD countries.

  2. you forgot coffee

  3. Stacey, Thanks for sharing the info, Really nice post. I Always wanted to understand and research on these, you helped me initiate. Thank you.

  4. Just a random thought. Enron was a pioneer in this.

  5. We need to be very careful about confusing metaphors with reality. We don’t consume “Internet” [sic] any more than we consume the letter “e” yet we accept a pricing model that treats bits as if they were kilowatts of electricity. In the same way “bandwidth” here is merely an accounting construct. It’s also important to recognize that the costs cited are also accounting constructs – the actual cost of a wire has very little relationship to the number of bits that travel the wire.

    The Internet is the way we use our resources, including legacy telecommunications facilities. It is not the wires and switches themselves. I go into this in more detail at my site. http://rmf.vc/NotSuper provides a simple example of the limits of the telecom metaphor.

    1. Sorry I have to disagee – the actual cost of a wire does have a very clear relationship to the bits that travel on that wire: the cost of the wire is spread out over its lifetime, say $10 a year. If you send 100 bits over that wire in a year, the transmission cost per bit is $0.01 and if you sent a 1000 bits over it then the cost per bit is lower by an order of magnitude, so on and so forth. Ideally you want to send as many bits as the wire can handle (maximum capacity utilization) to offset the fixed cost amortization of the wire. I hope you are not being dismissive about fundamental economics when you label these concepts as accounting constructs.

      1. I conditionally agree to part of this. I touched on this idea in my comments earlier. It seems logical to keep your circuit as busy as possible in order to maximize revenues from it. However, it seems that the monopolies have found it to be more profitable to pretend there is a bandwidth shortage, and to ration overvalued “data instruments” at astronomical prices. This often means the circuits are well under capacity, but the telco’s profits are maximized.

        The problem that I have with this, is that the telcos charge hundreds of times for access to the Internet, that it costs them to provide it. How that relates to “wire length” is that while it may cost $100 to install and maintain a cable to a metro location (close to IXP) and it may cost $1000 to install and maintain a cable to a rural home (far from IXP), by the time that wire has “worn out”, the maintenance expense of that longer wire does not justify the steep mark-up in the price of service. The maintenance expense is negligible in the long run.

        As before, the reason customers at the end of long wires pay more for their service is because they have fewer alternatives to choose from. It’s the *old* economy, not the “new” speculative one Stacey describes in the article. This new Internet economy runs on the principle of monopolistic gatekeepers speculating on how much profit they can make by maximizing their prices, meanwhile giving the public (and regulators) completely farcical explanations for the hikes to keep market pressures down.

        It seems you’re making the assumption that the cost of “longer wires” actually justifies the overcharging for Internet access, when it’s just a minor element that has been whipped into part of the telco’s smoke screen. If there is indeed a surcharge on bits to pay for the wire over it’s lifetime, then it’s buying enough wire to reach Mars, and customers are seeing none of the benefit. In reality, I think you’ll find it’s buying politicians.

  6. Good lord, Stacy. I wish you would try to be a real journalist for a change. This article literally worships the Enronization of the Internet instead of exposing this “economy” for the fantasy football game that it is. I understand this, Ron understands this, and dozens if not hundreds of your readers understand this. I’m sorry to sound so harsh, but your articles consistently demonstrate that you’ve swallowed the blue pill. I don’t doubt that some fatcats sit and trade packages of data based on some system of valuation they’ve agreed on, but the inputs don’t match the outputs, and the billable events are as absurd as they are arbitrary..

    I will say this now and I will say this forever: it costs virtually nothing to transmit megabytes. Trillionths of a cent per bit. The cost is in building out the network (which was paid for with public money long ago), expanding the network to keep up with demand (which is being delayed in order to create an embargo; a false shortage that drives up price), and keeping the existing equipment powered and maintained.

    Profit is is a matter of having the network serve as many paying customers as possible, at the highest rate per head. When billing by the byte, it may seem sensible to keep the network as close to available capacity as possible to maximize profit, but it seems to have turned out to be more attractive to simply charge customers as much as they will pay, and modify their perceptions with a false emergency. Somewhere on a chart, lines depicting the price of bandwidth vs the willingness of the public to pay, crossed at that point.

    If everyone turned off their modems for a day, the ISPs would not save any money, because their equipment is still using essentially the same amount of power, and their staff will all still be in place drawing wages. There’s a reasonable argument that they might save some money that day due to a dramatic drop in customer-generated service calls, but that’s not related to the quantity of data transmitted.

    Bandwidth is not manufactured, nor is it consumed. Bandwidth is a rate, not a quantity. The fraudulent “economy” you outlined exists only because of restricted access by providers, not due to significantly limited supply. Other factors you purport, like increased expense of Internet access and distance from IXP are more realistically described by increases in price due to lack of competition.

    You have your own backbone? Fine, we’ll trade data through the IXP at the price of, say, $10/gigabyte, but we’ll keep our traffic even, for a net cost of zero above the cost of keeping the equipment running. You don’t have your own backbone? Prepare to buy gigabytes from us at $10/ea, and resell it to your customers at 101% for a profit margin that keeps you out of the game. All legal. Too bad you can’t afford to run a cable to a cheaper IXP in another country.

    You see? It’s still about who has more influence to demand the price down. In the mean time, big telco-ISPs whimper about the “cost” of bandwidth when they don’t actually pay according to that scheme. It’s like how the USA has a very high tax rate, but in practice, loopholes and accounting gymnastics result in an extremely low actual tax rate.

    Noone should be counting bytes. The only fair and rational unit to charge bandwidth by is the capacity of the circuit over the length of the billing period. This will encourage ISPs to market faster connections for a premium price, and not ration packages of data as they do currently. People are hitting their monthly bandwidth caps in hours or minutes now! The circuits can obviously handle far more traffic. It’s all just to scare people away from Internet movie rentals and onto the ISP’s own IPTV services which don’t get billed by the byte at all. If phone lines and cables are substandard for fast connections, then ISPs will have incentive to bring them up to spec.

    Please spit up the kool-aid.
    Sincerely, A Fellow Austinite

    1. I meant to add one point to the end of the second paragraph above: “This cost is a tiny fraction of what ISPs bill for service, and is getting cheaper every year.”

    2. Stacey Higginbotham elfonblog Monday, November 5, 2012

      I’m not sure where you are getting this idea that I’m a fan of counting bytes (or even bits), but the point is that if you increase competition (and add IXPs to reduce distance) prices will and should drop. And yes, the costs associated with networks are the cost of the infrastructure plus the tiny bit of opex to keep the infrastructure humming. Not sure where you think I say otherwise — here or in my other stories.

      1. Respectfully, I get that idea from the consistent bias seen within your other stories, including the one that blithely opens by saying, to paraphrase, “since data caps are inevitable, let’s chat about coming to terms with a fair limit….”, heh.

        You could really win my respect by showing that you know the difference between what the industry wants customers and regulators to believe, and the reality as it is when stripped of metaphors treated as fact. Despite all of the points I raised, I’m disappointed that you chose to poke at the bit about bytes. I may not have a talent for brevity, but I would show you more respect than to ignore your points and just take a swat at straw men.

        To simplify my explanation one final time today; the cost of maintaining and providing Internet service is relatively miniscule and is dropping steadily. The PRICE of maintaining and providing Internet service is gigantic and growing, due to arbitrary fees and methods of valuation imposed by the top (monopoly) players, and the artificial shortage created for advantage, by their deceptive and *inappropriate* stock market model.

        The largest bill my ISP employers paid wasn’t for facilities, utilities, insurance, benefits or wages. It was the gargantuan bandwidth bill. Amazingly, all the rest was paid for with a modest markup to our customers, but it led to running a business with a razor thin profit margin. Though we were multi-hosted, our providers were unwilling to “trade” bandwidth with us. This would have nullified our bandwidth bill, but we would also have become genuinely competitive. They sabotaged our attempt to supply DSL the same way they did with every other alternative provider.

        I don’t begrudge an industry it’s profits, but when it’s profits are high enough to build a whole new Internet every few years, I say it’s high time to illuminate and empower the public. We depend largely upon the Press to do this. Our country has some of the slowest standard speeds coupled with some of the highest prices despite our abundance of “IXP’s” and the fact that we invented the Internet. No, actually the public is *not* satisfied with “fast enough” speeds nor proudly paying more due to our relative wealth!

        Perhaps my folly is in continuing to assume you don’t understand. It’s clear enough to some of your other readers who commented here. Even Bob Frankston and I have come to highly parallel conclusions about the Enronization of the Internet, though we arrive at it from different directions and employ different language. The burning question is: who are these experts you consult?

        Again, I urge you to be a journalist about this, and not take directly to heart what your industry friends and the 92-page treatises they promote say. You obviously have access to a considerable amount of information about this industry’s machination. What you do is create a vivid picture out of a hand-picked subset. The parts you omit would paint a radically different image. The industry is playing chess on a checkerboard, and it behooves you as a journalist to expose that.

  7. Samuel Satter Sunday, November 4, 2012

    Souns like a pretty solid plan dude.
    http://www.u-privacy.tk

  8. I’m sitting on a beautiful beach in Rocky Point, Mexico this morning reading your article. I stopped by Telmex yesterday to get pricing on DSL lines for my home, so I can have my own dedicated connection instead of sharing 6mb. of resort bandwidth with my 200 neighbors.

    Pricing is as follows:

    3 mb. = $389 pesos per month ($30)
    5 mb. = $599 pesos per month ($46)
    10mb. – $999 pesos per month ($77)

    This compares to my Comcast rate in Salt Lake City, where I get about 30mb. in a bundled package of voice, video, and data for about $200 per month, with the video package being the most expensive (multiple DVR’s, all the premium channels, etc.).

    No wonder Carlos Slim, the owner of Telmex, is the richest man in the world.

  9. Article has no match for title. Also it was not clear : “how does a country produce bandwth?”
    Using the term “produc seems a bit stretch.

  10. this is not a new topic. but it is nice to see a media outlet talk about something beyond the cheesy startup internet business

Comments have been disabled for this post