Reports have been filtering in during the last few days that consumers on Verizon and Time Warner Cable’s network are experiencing degraded service when they try to watch Netflix or YouTube videos. It may seem trite to whine about someone’s Arrested Development episode buffering, but the real issue is how big ISPs are trying to remake the agreements that underpin how the internet works.

As they do so, they are taking agreements that used to be negotiated by engineers based on web traffic and changing them into disputes negotiated behind closed doors in boardrooms, executives fighting over who has access to the end consumer. This could fundamentally change the way the internet works — making it more expensive to do business and erecting unnecessary barriers to innovation.

This is the new battle for the internet. The telcos and cable providers, intent on protecting their margins and their pay TV businesses, have taken network neutrality from the public world of consumer pricing and throttling to the data centers. Instead of banning Skype, or charging more for it on their networks, they want to change they way they charge content providers, demanding that they pay more for ports on the network when traffic starts filling them up.

The weapon in this battle is a concept known as peering. Peering is essentially an arrangement between two bandwidth providers — the companies that control the physical backbone of the internet — in which they send and receive traffic from each other for free. The logic is that the traffic sent from one network to another is reciprocated without adding extra costs and hurdles. This makes the web more efficient and redundant because companies don’t need to build out a network to connect every single service to every person who wants to consume that service.

There are about 50 major internet bandwidth providers around the world that connect to each other and work under that arrangement. There are about 5,000 other networks who then work with these major bandwidth providers to bring the internet to our homes, offices, iPads and iPhones. Of course, I’m being simplistic when I describe the notion of peering, but you get the gist. And this idea of peering — on which much of the modern internet is built — is coming under attack, thanks to ISPs like Verizon and Time Warner Cable that provide us (the consumers) with connectivity.

Two examples of this shift

Here’s how these battles tend to play out in the real world.

Picture a conference room in Arizona. In such a room last year, executives from Cable One, a cable broadband provider with 720,000 subscribers, asked the chief technology officer at one of world’s largest bandwidth providers to pay Cable One a fee each time it needed to add more ports to deliver content to Cable One’s end subscribers due to an increase in demand for certain types of content. The CTO of that bandwidth provider, who declined to be named, refused. He walked away rather than add a new, and to his mind, unsustainable, cost to the company’s bottom line. (A representative from Cable One did not immediately respond to a request for comment on this incident.)

Now consider another scenario, which is more in tune with how the internet currently works. Picture a data center in California, where Sonic.net, a Santa Rosa, Calif.-based independent broadband and phone company, also keeps its servers. Sonic.net, which has about 50,000 subscribers, sees a bit under 10 gigabits per second of Netflix traffic a day.

In order to deal with that traffic, Dane Jasper, the CEO of Sonic.net, has put a Netflix-designed box to cache Netflix content closer to the edge of the network where the ISP network meets the big bandwidth providers’ networks. This helps Sonic.net cut the cost that Netflix traffic can impose on its operations. Essentially, Netflix pays for its traffic to travel across most of the network before dropping it at Sonic.net’s door.

Telcos want to make the internet like the phone system

Open Connect, Netflix’s content delivery network, is a new wrinkle in the old idea of networking peering. But it tries to keep the same relationship, namely that both parties (those sending and those receiving content) share in the costs of delivering traffic to the end consumer. This model has worked for years, but some ISPs want to change it.

They see their pipes becoming commodified — dumb, as it were. So they want to build a new internet model: one based on terminating traffic that looks a lot like the old-school telephone networks, where ISPs serving the last mile can charge content companies and bandwidth providers for upgrades they make to the network in order to carry the increase in traffic. The point of connection and negotiation has become the ports where traffic hops from the bandwidth providers onto the ISPs’ network.

An OECD report on peering and internet exchange points from last October explained that because many areas of the world have efficient and cost-free peering relationships, the prices for data are around 100,000 times lower than the price of a voice minute.

As consumers, we are experiencing this underground tussle via poor video streaming and what are effectively throttled connections. For example, earlier this week Om and I wrote about Verizon’s refusal to work with Cogent because it was carrying traffic for a large video provider. Verizon was letting its connections to Cogent (multiple 10 gigabit per second ports) run hot — a euphemism for getting crammed with traffic. And that’s because Verizon has refused to open more ports — the equivalent of opening lanes of traffic on a highway — to Cogent.

Normally when this happens the two parties come to some sort of agreement about adding another port, and then the problem is solved. But in this case, Verizon doesn’t want to play. We don’t have details about what’s happening in the Verizon/Cogent spat, but we can share how these fights have played out in the past, thanks to a public peering squabble that happened in 2010.

We’ve been here before

On the final weekend of November 2010, engineers at Comcast were stunned to find themselves overloaded on ports that connected to Level 3 Communications. These ports — where Level 3 terminated its traffic destined for Comcast’s last mile subscribers — were getting slammed with 10 to 100 times the bits they normally saw. And since overloaded ports result in a crappy experience for the end customer, Comcast needed to figure out what was happening.

Engineers trying to determine where the huge influx of traffic was coming from finally settled on the culprit — a deal that Level 3 had signed earlier that month to carry traffic for streaming video service Netflix. At the time, Netflix traffic made up 33.7 percent of North American web traffic, and in that one weekend most of that traffic suddenly came into Comcast’s network from Level 3 as opposed to Akamai, Netflix’s previous content delivery partner.

The internet is flexible and redundant, but it’s still grounded in actual physical infrastructure. So when all those bits suddenly came into Level 3′s ports instead of the ones Comcast had designated for Netflix/Akamai traffic, packets started dropping and Netflix customers on Comcast network experienced poor service. It’s not all that different from when the police shut off access to a major highway and send people onto local streets: Traffic backs up and everyone’s unhappy.

Luckily, building new roads on the Comcast network isn’t as labor-intensive as building out roads in real life. Even so, when it came to adding new ports for Level 3, Comcast took the opportunity to try to renegotiate peering contracts with Level 3. “These fights over peering can be like third-graders squabbling on the playground,” said Patrick Gilmore, chief architect, network infrastructure for Akamai, in an interview last summer.

While the fuss over the Comcast and Level 3 spat eventually died down after the two parties agreed to some form of give-and-take on getting new ports on the Comcast network, it highlighted a basic fact of life on today’s internet. In many cases, as the big get bigger, the internet’s core weakness isn’t technical; it’s the fact that the main players are now so large that they are in the midst of negotiating a new balance of power.

It’s an IP version of the treaties and shifting alliances in Europe in the late 1800s that then fell apart during the first World War. And while lives aren’t exactly at stake in these peering battles, the outcome of these fights might change the way the internet works, making it more expensive to build a business on the internet and allowing ISPs to become a new gatekeeper.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected consumer first-quarter 2013: Analysis and outlook
• Who and what to watch in the new era of the living room
• Netflix may suffer from limited mobility

First up, Amazon announced it’s added “more than a thousand books, games, educational apps, movies and TV shows to Kindle FreeTime Unlimited,” its monthly subscription streaming offering for kids. Not surprisingly, Amazon didn’t provide subscriber numbers — though Kindle VP Peter Larsen said “the vast majority of our customers who take advantage of our one month free trial of FreeTime Unlimited choose to subscribe and enjoy our content library built just for kids.”

Of course, Amazon’s announcement refers to the Viacom shows that it grabbed exclusive rights to after Netflix lost them: “Among the content on FreeTime Unlimited are “exclusive Nick Jr. favorites from Viacom such as Dora The Explorer, Go, Diego, Go!, Blue’s Clues and The Backyardigans.”

Then Netflix announced that it’s launched a separate section of its site called Netflix Families, which “helps parents with the daunting task of entertaining kids this summer.” The site offers family recommendations in different categories (“family movie night,” etc.) as well as videos from parents about how they are “embracing the ease and flexibility of Netflix.” And Netflix notes that “Netflix works on almost any device with a broadband, WiFi or mobile data connection” — while Kindle FreeTime Unlimited is only available for the Kindle Fire.

Photo courtesy of Shutterstock / jwblinn

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected consumer third-quarter 2012
• Q4 Wrap-up: SOPA and the future of digital content
• What the shift to the cloud means for the future EPG

Netflix just posted the tool, which provides a birds-eye view of its own cloud landscape (cloudscape?), onto its Github page, the last of a series of open-sourced Netflix goodies to go up.

Asked why Netflix went its own way with an AWS monitoring tool, Ariel Tseitlin, director of cloud solutions for Netflix responded by email to say:

“We built Ice to to give us deep insight into our cloud usage that we couldn’t find with any of Amazon’s or 3rd party offerings.  It gives us the visibility and operational support to manage our mature complex environment and we hope that the rest of the cloud community can benefit from it.”

Like the other AWS checkers, Netflix Ice relies on data supplied by Amazon’s own APIs. With those stats in hand, it tracks usage by accounts; regions; type of service (EC2, S3, EBS); and usage types by instance size. It also tracks whether you use on-demand, reserved or other instance types, the pricing of which varies. Given the sheer number of services and options AWS offers, it’s clear that tracking all of that is a handful, especially for a large organization.

Ice is a Grails project consisting of a processor, a reader and a UI. The processor takes in the Amazon billing file and makes it available to the reader, which renders it to the UI. The UI queries the reader and renders interactive graphs and tables for the browser.

Netflix has years of institutional knowledge on this topic, relying as it does on Amazon infrastructure.  As the blog post states:

Netflix is a highly decentralized environment where each service team decides how many resources their services need.  The elastic nature of the cloud make capacity planning less crucial and teams can simply add resources as needed.  Viewing the broad picture of cloud resource usage becomes more difficult in such an environment.  To address both needs, Netflix created Ice.

So, if you’re a big AWS shop and would really like to get a handle on just what all your developers have running (or deployed and not running) on AWS cloud, you might want to check Ice out.

Los Gatos, Calif.-based Netflix is really trying to propagate a range of the tools and utilities it uses to make sure its streaming media empire runs well on AWS. Most recently, it posted Isthmus, which vows to manage elastic load balancing across AWS regions. Isthmus, in turn, builds on Zuul, another tool that acts as a gatekeeper between Netflix’ own API and other Netflix services and AWS Elastic Load Balancer that routes video to users. The goal is to prevent meltdowns like the one that hit Netflix last Christmas Eve. 

Netflix, as its cloud guru Adrian Cockroft said at an open source open house it hosted a few months ago, really wants outside companies to deploy its components and, it’s very interested in getting other, non AWS cloud vendors, to deploy these tools as well. One can only imagine why. Cockcroft will be joining us this week at Structure in San Francisco so come by to see what he has to say.

Update: Now that Amazon and Netflix have joined a half dozen or more third-party AWS monitoring tools in the pool, I’d  love to hear from some of the competitors in this space — folks, what does Netflix’ move mean to you and your AWS monitoring tools? Use comments to respond.

This story was updated at 4 p.m. PDT with additional competitors in this space and a request for comments.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

Peering is essentially an arrangement between two bandwidth providers where they send and receive traffic from each other for free. The logic is that the data sent from one network to another is reciprocated. Verizon runs one of the largest last mile networks and owns the descendants of MCI. Cogent is one of the largest bandwidth providers, and its network is spread across the globe in hundreds of cities.

Cogent and Verizon peer to each other at about ten locations and they exchange traffic through several ports. These ports typically send and receive data at speeds of around 10 gigabit per second. When the ports start to fill up (usually at 50 percent of their capacity), the internet companies add more ports. In this case, through, Verizon is allowing the ports that connect to Cogent to get crammed. ”They are allowing the peer connections to degrade,” said Dave Schaffer, chief executive officer of Cogent said in an interview. “Today some of the ports are at 100 percent capacity.”

“Think of it as the on-ramp to the freeway being log-jammed,” Shaffer said. And that means your Netflix content, especially content sent by Netflix’s content delivery network, slows down, and you get pixelated pictures and buffering.

While not naming Netflix directly, Verizon has indicated to Cogent that the reason behind its actions is that Cogent is moving traffic for a large video provider. Schaffer confirmed the Netflix is one of their largest customers. “Over the past year Netflix has become a big partner for us. This is a business model problem, not an engineering problem,” Schaffer said.

Our sources tell us that Netflix recently bought 2 Terabits of bandwidth capacity in part to get around such cramming that was happening in places where it sends traffic directly to certain internet service providers.

When we called Verizon about this story asking if Verizon was having a problem with Cogent over peering issues associated with Netflix, Verizon spokesman Bill Kula said he’d get back to us. A few minutes later he sent the following reply that didn’t answer our question:

Verizon operates one of America’s lowest-latency, highest capacity networks. The various classes of Internet speeds we offer are among the fastest in the nation. Time and again, customers rate us best in class in various reports and surveys. Our customers enjoy a consistently superior Internet experience because our networks can adapt and grow with their use.

Netflix has been growing like crazy and it now accounts for a whole lot of Internet traffic — almost one out of every 3 bits (32.3 percent) sent downstream to users in North America is Netflix traffic according to Sandvine, a company that makes traffic monitoring gear for ISPs. That’s a lot of congested ports.

Netflix’s growing popularity has made it a target of ISP (internet service providers) vitriol and anger, especially those who offer competitive services. Verizon, for instance owns 50 percent of Redbox, a video-over-the-Internet service that is competitive with Netflix. Time Warner Cable and Comcast are other large providers that has allowed degradation of the online video experience on its networks — after all the logic is that as people start to have a bad Netflix experience, they start to look for alternatives — perhaps the ISP’s own pay TV offering.

This isn’t the first application last mile network operators have tried to degrade — last year the wrath of the Baby Bells and cable companies fell on Megaupload, a file sharing company started by Kim Dotcom, Schaffer said. That too was one of the big bandwidth-hungry services popular with the end customers of the ISPs — actual consumers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• OTT technologies and strategies for broadcasters
• What the shift to the cloud means for the future EPG
• Who and what to watch in the new era of the living room

Time Warner Cable felt compelled to publish the post after a number of customers had mused in recent months that the ISP was behind the sub-par performance of their YouTube videos. The supposedly damning evidence? Users had discovered that blocking a few IP numbers vastly improved their experience.

Time Warner Cable’s blog post doesn’t go into the IP number blocking, but explains the core of the problem this way:

“The internet is not as simple as one wire connecting a website’s servers to a customer’s home. Traffic originates in countless places, heading toward billions of end-user destinations. Each network that carries web traffic is itself a collection of a number of complicated technological and business relationships. As traffic flows from one area of the internet to another, it passes through this network of technologies, agreements and protocols and culminates in each particular user experience.”

In other words: It’s all about peering, and where in the network content is cached — which, as Time Warner Cable suggests, is as much about business as it is about technology. Case in point: Netflix has been trying to get ISPs to join its own Open Connect content delivery network and install its OpenConnect caching servers on ISPs’ networks.

The video service is arguing that this would greatly improve customers’ streaming experience while keeping down costs for ISPs, and Netflix is even willing to install and maintain these machines for free. However, getting U.S. ISPs to join has proven difficult. Not only are companies like Time Warner Cable and Comcast operating their own TV services, which makes them Netflix competitors, but they also want Netflix, Google and others to pay for distributing so much traffic on their networks.

Netflix has started to take this peering fight to the court of public opinion. The company is publishing a monthly ISP speed index, and launched a new Super HD video quality level that is only available to customers of ISPs that have joined Open Connect. Time Warner Cable has responded to this by alleging that Netflix is discriminating against its customers – a charge that Netflix unsurprisingly denies.

Google has thus far kept quiet in this peering fight. But the fact that its users resort to blocking IP numbers to get better quality levels on certain ISPs, as well as Time Warner Cable’s new denial, makes it clear that even for YouTube users, the peering wars are far from over.

Photo courtesy of Shutterstock user Swetlana Wall

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What the shift to the cloud means for the future EPG
• Who and what to watch in the new era of the living room
• Connected Consumer Q1: The Over-the-Top vs. Pay TV Battle Heats Up

The shows will begin running in 2014, and though Netflix didn’t release more specifics on content, it seems clear that most of them will be aimed at kids. That means the DreamWorks deal could help replace the Viacom shows like Dora the Explorer and SpongeBob SquarePants that Netflix recently gave up (and Amazon then grabbed).

This isn’t the first deal that Netflix and DreamWorks have made: The two first signed a deal in 2011 and then, this past February, announced an original kids’ show that will air in December.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

In a Friday blog post, Netflix’s Ruslan Meshenberg explained that Isthmus manages Elastic Load Balancing services in multiple regions in order to keep latency low for users in the event that ELB goes down in one region. On Christmas Eve, the issue was that state data got deleted, causing issues for the control plane tasked with managing load-balancer configuration and bringing some down some ELB load balancers. Now that sort of error could be less likely to affect Netflix service.

Source: Netflix

Isthmus builds on the newly open-sourced Zuul tool for managing the Netflix API and Elastic Load Balancer services that my colleague Barb Darrow wrote about earlier this week. “Zuul is at the core of the Isthmus setup — it forwards all of user traffic and establishes the bridge (or an Isthmus) between 2 AWS regions,” Meshenberg wrote.

Now, Netflix can split user traffic evenly between a US-East region and a US-West region. It’s testing to see if production-scale traffic can be shifted to one region in the event of another outage.

Netflix constructs a lot of tools to make applications running on AWS work better, and Zuul and Isthmus are just the latest. Netflix Cloud Architect Adrian Cockcroft will discuss some of them at our Structure conference next week, as well as the Open Connect content-delivery network Netflix built using its own custom hardware.

Presumably, once Netflix makes Isthmus available as an open-source service, other AWS customers could adopt it and finagle it to fit their own deployments. Then again, AWS might adjust itself accordingly. One would think that would be in Amazon’s best interest as it strives to gain more enterprise customers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

But there may be an unintended consequence of the new profiles: even better service for people who are illegally sharing passwords.

Of course, anyone with a busy household on the same streaming account would know that over time, finding a good movie or picking up where you left off on a TV show can be a major hassle.

By neatly organizing your family members into separate user profiles, there’s less mixing of tastes and histories — the media equivalent of keeping your Play Doh colors separate to prevent it from becoming a dried-out mass of marbled colors. You won’t have “Gut-Wrenching Horror Film” recommendations next to “Early Learning Show” titles anymore, and that small change will make a big difference in the general convenience of finding, streaming and enjoying media.

The same thing will happen when a person with a Netflix account lets half-a-dozen friends skim off for free — something plenty of 20-somethings do regularly. With the new mulitple profiles, it’ll be even easier for freeloaders to piggyback on someone else’s account. Users will be able to dole out passwords to their friends, set up user accounts, and then live happily on one subscription with separate histories and recommendations. It’s a new level of convenience for practically no money. Netflix CEO Reed Hastings has said that he doesn’t think password sharing is that big a problem.

But maybe, just maybe, this user profile feature is the perfectly laid trap — the carrot positioned pristinely in the middle of an iron-jawed clamp called unauthorized access. Depending on what kind of data Netflix tracks between accounts, even a little thing like logging IP addresses could go a long way toward programming a bot that shuts down accounts with too many foreign connections.

Technically, illegal users could spend a year in jail for violating the Computer Fraud and Abuse Act, which can lay the hammer down on password distribution for “exceeding authorized access.” Of course, it’s highly doubtful that Netflix or any other online media distribution company would take the time to prosecute such a thing. A terse account shutdown would be enough to dissuade users from skimming.

Netflix’s decision is a smart one, whether it’s a true convenience or the best piece of technological flypaper to catch rampant password distributors.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected consumer first-quarter 2013: Analysis and outlook
• How consumer media will change in 2013
• Takeaways from connected consumer’s second quarter

The latest is called Zuul, a nod to the gatekeeper in Ghostbusters and, like its namesake, Netflix Zuul acts as a gatekeeper between the Netflix API (and other Netflix services) and AWS Elastic Load Balancer that routes video to users. (see chart.)

Netflix API lead Daniel Jacobson explains it better than I could: “Zuul is basically a front application for the Netflix streaming application… it does much of the system-level work such as dynamic routing, load shedding, insights, health analysis etc so the other backend applications in the system can maintain focus on their part of the streaming application,” he said via email.

Zuul is applicable for any application that handles external network calls — it could “front” an international web site that needs to route traffic to different data centers or cloud regions based on geo information, Jacobson added. For Netflix, Zuul is used to handle HTTP transactions around metadata, not the streaming video itself. In that way, it just deals with typical web traffic for web sites and apps, he said.

Netflix outlines Zuul further on its blog.

Zuull gives us a lot of insight into our systems, in part by making use of other Netflix OSS components. Hystrix is used to wrap calls to our origins, which allows us to shed and prioritize traffic when issues occur. Ribbon is our client for all outbound requests from Zuul, which provides detailed information into network performance and errors, as well as handles software load balancing for even load distribution. Turbine aggregates fine­grained metrics in real­time so that we can quickly observe and react to problems. Archaius handles configuration and gives the ability to dynamically change properties.

When it comes to delivering video — or potentially other services to customers — there are a lot of moving parts. As the Zuul post attests, new AWS regions get deployed, new catalogs are added for new countries, new services come online. To handle those changes, the edge service helps enable faster and more flexible deployments.

Zuul will join other Netflix tools including Eureka, a mid-tier load balancing tool, and the popular Simian Army which includes the Chaos Monkey.

Netlix is championing the use of its open source tools and services big time. Last February it hosted an open house  at its Los Gatos, Calif. headquarters for open source developers to talk up tools to come including many more monkeys.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What Amazon’s new Kindle line means for Apple, Netflix and online media
• How the mega data center is changing the hardware and data center markets
• How devops can reduce cycle times

In many ways it would seem that the rise of gigabit networks would crush the business of providing pay TV, but in fact, if cable companies play it smart, they may find a way to walk the line as their industry transitions to all-IP content delivery over broadband networks. They may even find new sources of revenue by offering IP services such as home security and automation. To understand what cable firms are dealing with, I spoke with Phil McKinney, the president of CableLabs, the industry standards setting body that is responsible for pushing cable’s access technologies.

CableLabs is the organization behind the DOCSIS 3.0 standard, which has helped cable companies roll out 100 Mbps and faster speeds. Unfortunately, those speeds have a practical limit that won’t help cable providers like Comcast or Time Warner Cable compete with Google’s gigabit networks. And if AT&T or municipalities get aggressive about deploying such networks, cable providers might find themselves selling the equivalent of feature phones in a smartphone world.

Getting cable to a gig

Enter DOCSIS 3.1, the next generation of the cable access technologies. The new standard will allow cable firms deploying D3.1 equipment to deliver up to 10 gigabits per second down and 1 gigabit up. The technology uses OFDM technologies familiar to the wireless industry to cram more bits into a single megahertz of available spectrum used in the cable plants (it’s 11 bits per hertz if you care). Thus, cable providers can then deliver more bandwidth using their existing radio frequencies.

These RF channels are part of cable’s legacy of delivering analog television signals over coaxial cable. In today’s hybrid fiber and coax networks some of the overall transmission is digital, but the coaxial and RF frequency limits remain in some parts of the network.

Cable firms still haven’t gone all-IP, which means that most cable companies are dedicating some of their spectrum to their pay television business and some to delivering broadband. One technology uses IP and the other uses QAMs. But as people demand more bandwidth and higher definition TV channels, cable operators must decide where to allocate their limited spectrum, or lose market share they have gained in the broadband market.

McKinney is also touting new compression codecs like HEVC that help lower the number of bits in a stream but still deliver high-definition quality. It uses half the information that MPEG-4, the current standard, uses. That gives cable companies a little more room on their spectrum to allocate for more broadband channels or more TV channels. McKinney notes that CableLabs is moving faster than it has ever moved in order to get DOCSIS 3.1 out to constituents — achieving in two and half years what it took five to do for previous standards. Comcast says it expects to start deploying DOCSIS 3.1 in 2015.

But what about the business model?

And speed is important, because widespread access to high-speed broadband is threatening the cable industry’s core business — packaging a bunch of channels together and selling it to end consumers, as well as selling some advertising against those channels. On one side there are people cutting the cord — canceling their subscriptions and relying on content from Netflix, Hulu or even just over-the-air broadcasts. On the other side are content companies pushing for higher fees from cable operators, especially for things like live sports, which many analysts believe are the main reason people don’t dump their cable packages altogether.

But the cost of cable is rising, something consumers are fighting as they become more accustomed to picking and choosing their own content on demand. To satisfy those consumers cable companies are offering their own IP-delivered services that bring on-demand content to subscribers’ tablets and phones, even when they are outside their home.

That embrace of technology though, can require tradeoffs for cable providers. For example, Comcast now delivers all of its video on demand content via IP, which means it divides its available spectrum into three chunks. One is for the traditional cable TV that’s broadcast, one is for broadband and one is for delivering the bandwidth for its IP-based Xfinity VoD service. AT&T has done this with its U-Verse services on its copper lines, but Comcast got in trouble for it last year when people questioned if that practice violated network neutrality, since Comcast doesn’t count its Xfinity content as part of its bandwidth cap.

Recouping lost revenue in an all-IP world

With DOCSIS 3.1, Comcast may have more headroom to raise its caps if its network is truly congested at the cable plant, but the business challenge remains. It must also figure out how to keep customers from dumping a $200 monthly charge for both TV and broadband and choosing instead a $50 broadband package. Adding faster speeds and charging more for those speeds might be one way to keep revenue up. And despite cable industry fear-mongering about upgrade costs, McKinney estimates that the upgrade to DOCSIS 3.1 gear should cost cable companies less than the upgrade to DOCSIS 3.0, which analysts put at roughly $100 per home.

Chart courtesy of Stifel.

But aside from charging more for better broadband, cable companies shouldn’t have to give up on pay TV. Already companies like Time Warner Cable are experimenting with cheaper programming bundles in additional to concessions like allowing customers to watch any show, anywhere, on any device. The pay TV providers already have relationships with the content companies, and while they may not be the only path to mass market anymore for the Disneys and HBOs of the world, they still are an important channel.

Cable companies have tools they can use to protect content, they still have relationships with more than 80 percent of the U.S. households and they are aggressive about offering content in a way that consumers want. So, if they can transition to more of an a la carte option, using IP to deliver those choices on demand, they could still provide a service that consumers are willing to pay for. And thanks to new standards described above, the bandwidth is there to do this.

So cable providers just need to walk the line between cannibalizing their traditional pay TV business with IP-delivered services, while upgrading their networks to ensure they can still deliver a quality experience while maintaining their revenue and profits. The big telcos walked this line a few years back when they had to transition people from wireline networks to cellular service without hurting their own profits and revenue.

The cable business is a little tougher because they have the content companies in there demanding more money and seeing new avenues for distribution, but as disruptive as this shift is, I think in a few years we might see an even bigger one once pay TV providers realize they can take all of their content and deliver it over the top. Maybe the internet won’t eat the cable company after all.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• ESPN Leads the Way Over the Top, But Will Others Follow?
• What’s so bad about being a dumb pipe?