Comcast to Install Speed Bumps for Bandwidth Hogs

31 Comments

Updated: Comcast has filed its plan with the Federal Communications Commission detailing how it intends to govern traffic on its network, and says it should affect less than 1 percent of its users. It will start to go live commercially as of Nov. 15, and will be implemented  throughout Comcast’s network by the end of the year. As expected, the plan hews closely to what Om laid out back in March in a piece he wrote after sitting down with Comcast CTO Tony Werner. Essentially, folks using a lot of bandwidth at any one time on a crowded network will see their traffic slowed temporarily. The management will affect uploads and downloads and will be protocol agnostic.

The cable company was ordered to file such a plan last month, after the FCC censured it for throttling peer-to-peer traffic on its network. The FCC determined that the Comcast network management was problematic because it was done without informing subscribers and targeted a type of traffic that might be considered a competitor to Comcast’s cable business. Comcast maintains it did nothing wrong and was trying to maintain a good user experience on its network.

That will comply with the letter of the FCC order (which Comcast is appealing), but we had some additional questions of our own, as we detailed a few weeks ago in our post on the topic. Comcast answered our worry about bandwidth caps with its announcement of a 250GB-per-month limit. Judging from the information on its site about the new plan, Comcast answered all but our questions about who is providing the equipment to enable it’s efforts. Update: Sandvine will be providing some of the equipment as will Camiant. The plan will affect both uploads and downloads whenever the network is congested as detailed below:

Assuming that is the case, customers’ accounts must exceed a certain percentage of their upstream or downstream (both currently set at 70%) bandwidth for longer than a certain period of time, currently set at fifteen minutes.

A significant amount of normal Internet usage by our customers does not last that long. For example, most downloads would have completed within that time, and the majority of streaming and downloading will not exceed the threshold to be eligible for congestion management. And the majority of longer-running applications, such as VoIP, video conferencing, and streaming video content (including HD streaming on most sites) will not exceed these thresholds either.

All in all it looks pretty neutral like it may cause objections, but we’ll be digging through it a bit more and I’ll update the post with more in the next few hours. Update: Theoretically it would be possible to cut off a person’s traffic using this method if the entwork were incredibly congested for a long amount of time. From the filing:

NetForecast, Inc. explored the potential risk of a worst-case scenario for users whose traffic is in a BE state: the possibility of “bandwidth starvation” in the theoretical case where 100 percent of the CMTS bandwidth is taken up by PBE traffic for an extended period of time. In theory, such a condition could mean that a given user whose traffic is designated BE would be
unable to effectuate an upload or download (as noted above, both are managed separately) for
some period of time.

For those who want to check it out on their own, below are links to download the files.

Comcast’s Current Network Management Plan

Comcast’s Future Network Management Plan

Comcast’s Compliance Plan

31 Comments

Travis

I am a comcast customer and yesterday I seen some fatass climb on my telephone pole, and when his fat ass came down my ping from the online game I play went from 26 to 51, i will definitely be dropping comcast for DSL!

Charles Richey

I would definitely be changing service. Here in Las Vegas Cox communications has a strangle hold on high speed service (dsl really doesn’t count). Thankfully they haven’t started down the same road but prices have risen steadily over the last 2 years.

bmoura

>>optionshiftk said:

I am very fortunate that my ISP(cablevision) is not capping my bandwidth. If I were a comcast subscriber I would be running to FIOS or nearly anything else that beats out comcast.<<

You are very fortunate! Here in Silicon Valley, Verizon FIOS has yet to be offered. So all home Internet users have to choose from is Comcast and AT&T – both capped.

Jason Livingood

Joel Strellner asked: “Are the limits applied based off of what my cable modem states, or at the location that everything is joined in my neighborhood? There are times where Comcast’s network speeds are clearly limiting me, which I think is due to my neighbors.”

Both. First the upstream or downstream port you are served from on the CMTS must be over the threshold. Then, user traffic can be considered.

“Also, how is the sampling done? Is it constant, or just a sample every 15 minutes? For example, if I download a large PDF that takes 1 minute and uses 90% of my connection – exactly 14 minutes later, I do another which lasts two minutes and uses 90% of my connection. Would I be throttled even though I did nothing in the 14 minutes between those two downloads?”

The sampling is continuous, and we look at the average over the period.

“Will Comcast have a way for me to see if (when) I was throttled, or how much of my 250GB monthly limit I have used?”

Not yet – but it is on the list for new features to be developed soon (in the short-term we’re more focused on deployment). I have a bunch of stuff in the longer-term list of things to do, such as:

– Work on methods to put QoS controls in the hands of users, so that a user could configure their own traffic to set their preferences. For example, some users may want to prioritize their VoIP traffic, while others may wish to prioritize online gaming.

– Work on methods to expose near-congestion and congestion state information to applications and users. For example, this could enable a user to know if their traffic is temporarily in a lower priority, or was at one time in the past, or it could enable their application to automatically adjust to such network conditions based upon this information.

Jason
Comcast
National Engineering & Technical Operations

Joel Strellner

Are the limits applied based off of what my cable modem states, or at the location that everything is joined in my neighborhood? There are times where Comcast’s network speeds are clearly limiting me, which I think is due to my neighbors.

Also, how is the sampling done? Is it constant, or just a sample every 15 minutes? For example, if I download a large PDF that takes 1 minute and uses 90% of my connection – exactly 14 minutes later, I do another which lasts two minutes and uses 90% of my connection. Would I be throttled even though I did nothing in the 14 minutes between those two downloads?

Will Comcast have a way for me to see if (when) I was throttled, or how much of my 250GB monthly limit I have used?

Daniel, The Hosting Blog

I heard about their plan to limit all the users to 250GB because of service abuse. I know of some people who will logon and constant download tons and tons of crap just to do it. Like this guy that I know runs a warez site and he says every time he logs into a network he will set up his download manager to cap out a line which in turn ruins service for the rest of the neighborhood.

I think most people will be fine with 250GB a month. Even avid gamers should be OK at this limit. But if you are gaming and downloading 10 movies in the background you might be in trouble

Jason Livingood

Robb said: “Customers may not like what they learned, but these clarifications are good things.”

I think you are right on the money that these (and other recent) clarifications are a good thing. Transparency and openness are great things, and over time the average customer will have a much better understanding of the mechanics of our – and other companies’ – ISP services. I think we can already see where Internet users are becoming much more savvy and educated about related topics, like the privacy of their web surfing habits, security of their personal data, protection from malicious software, software EULAs, etc.

Regards
Jason
Comcast
National Engineering & Technical Operations

Robb Topolski

–Quote–
I believe the way we sell service is that we offer burst-to speeds when you sign up for a service tier, not a Committed Information Rate (CIR) of a guaranteed 24×7 constant speed.
–EndQuote–

I think Comcast has taken strides in recent months that have clarified the parameters of its product, but considering the ferocious reaction to the disclosure of an 250 GB “Excessive Use” amount, I think its clear that a sizable part of the customer base had a different understanding about the nature of the service that they had subscribed too.

Customers may not like what they learned, but these clarifications are good things.

Richard Bennett

Griffon, you’re already paying much less than 70% of the bill for a Committed Information Rate. A dedicated T1 – 1.44 Mb/s – runs over $400/mo most places. Nobody’s stopping you from ditching your cable modem for a deal like that.

Jason Livingood

Griffon said: “Great, dose that mean I only need to pay them 70% of my bill since that is all the bandwidth they will guarantee me perpetually.
God I wish we had any real competition in this assbackwards market.”

I’m not a marketing or legal person, but I believe the way we sell service is that we offer burst-to speeds when you sign up for a service tier, not a Committed Information Rate (CIR) of a guaranteed 24×7 constant speed. But more to the point, the 70% threshold in the new congestion management system is not a limit in any sense, as you can easily go beyond it. It is used, once an overall access network utilization threshold is crossed, to tag some packets with a lower QoS for a short period of time. Up to the point of actual congestion, there is no effect of this and packets are processed normally.

Jason
Comcast
National Engineering & Technical Operations

Griffon

Great, dose that mean I only need to pay them 70% of my bill since that is all the bandwidth they will guarantee me perpetually.
God I wish we had any real competition in this assbackwards market.

Jason Livingood

Greg asked: “Maybe I missed it, but are the 70% thresholds for the individual cable modem connection or higher in the stream?”

If you pull up the FCC filing about the new technique, you can take a look at page number 11. On that page, there is a visual decision flow chart, using the example of an upstream port that you may find a good way to better understand the technique.

Basically, for upstream, a port must be over 70% for over 15 mins. Then, you go to the next step, which is to analyze recent user traffic consumption. For downstream, the port % is 80%. In both cases the user % is 70% for over 15 mins.

I’d fully expect to need to fine-tune some of these settings over time, as we learn more from the system as it is deployed more widely.

Hope that helps…

Jason
Comcast
National Engineering & Technical Operations

Jason Livingood

Nick said “…if it effects less than 1% — that says a lot.”

Thanks, Nick. We’ve been pleased with the data collected in the trial markets so far. The average percentage of customers that had their traffic managed on any particular day ranged from 0.09% to 0.60% in our trial markets, and there were some days when no subscribers were managed at all.

Jason
Comcast
National Engineering & Technical Operations

Richard Bennett

Good answer, Jason. It’s amazing that so many people are so naive about network congestion after all that’s been written about the subject in the last year.

People seem to accept that the sewer system can’t handle everybody flushing their toilets at the same time, but somehow the Internet should handle the same scenario without a hiccup.

As if.

Jason Livingood

Eyebee —
“I still think it sucks. It’s about time the networks were built up to a specification that allows not only for present usage but near future usage too. More and more data of all kinds is going to be transmitted via the Internet, much of which isn’t illegal P2P either.”

Unfortunately, congestion can occur in any IP network, and congestion feedback is a part of existing Internet standards. There are examples in Japan and other countries with 100Mbps service, where congestion can still occur at some point on the network. Our network has a shared access network, like many wireless networks, which basically means it is shared one hop sooner than some circuit-based networks. Those networks often see congestion sometimes at their aggregation points. Nevertheless, I’m sure you have heard all that before. ;-) We’re also upgrading to the new DOCSIS 3.0 standard, which adds a lot of capacity and speed to our particular network.

“I watch video, stream audio (internet radio stream). Comcast wants to dissuade its users from these kind of activities…”

Far from it – those kind of activities are the future. I think we’ll see more and more of that kind of traffic, as well more “unattended” applications and “ambient” apps that are always chirping away in the background.

Regards
JL
Comcast

Greg

Maybe I missed it, but are the 70% thresholds for the individual cable modem connection or higher in the stream?

I’m concerned that my online backup process could take even longer than it already does, if the former case is true.

Eyebee

I still think it sucks. It’s about time the networks were built up to a specification that allows not only for present usage but near future usage too. More and more data of all kinds is going to be transmitted via the Internet, much of which isn’t illegal P2P either.

I watch video, stream audio (internet radio stream). Comcast wants to dissuade its users from these kind of activities as they want subscribers to sped huge amounts of money on ever more premium rate TV channels.

With ISPs with an attitude like Comcast, it’s no wonder the US is 16 or 17th in the world league table for Internet speed and reliability.

Jason Livingood

We have two risk scenarios depicted that we’ve tried to address. Robb already asked this Q elsewhere, so I am posting the same response here for your information.

1 – What is oscillation, and how is it prevented or minimized in the new method?

Oscillation refers to the situation where a cable modem is assigned to a lower Quality of Service (QoS) priority (Best Effort, or BE) after one measurement interval, then promoted back to the higher QoS priority (Priority Best Effort, or PBE) in the next interval, just to be assigned to the lower priority again in the following interval. We refer to this up and down assignment as oscillation.

Part of the new congestion management system is the use of thresholds to determine when a cable-modem is in a high consumption state and when it is not. At the start of the trial the congestion management algorithm used a single threshold to determine if a cable-modem was in a high consumption state or not. If a cable-modem’s utilization was just over the threshold in one time interval and it was then put in the lower priority, it is very possible that because of the lower priority its use will now be just below the threshold and so it will be promoted back to the higher priority in the next interval.

To resolve this issue, we instead implemented two thresholds. The higher threshold determines when a cable-modem is put into the lower priority class (BE). The lower threshold determines when the cable-modem can be taken out of the lower priority class and restored to normal priority (PBE). By having the two separate thresholds, the chance of oscillation occurring is greatly reduced.

This kind of oscillation is often found in control systems that use a single threshold for both on and off states. A standard approach to reduce rapid cycling through the states is to add a second threshold which gives the system a range of control. The range between the two thresholds is called hysteresis. Most thermostats that control residential heating systems operate this way, for example, and many other control systems have oscillation-prevention features as the Comcast system does.

2 – What is user starvation and how is it prevented or minimized in the new method?

User starvation was an extreme case identified with theoretical or simulation models of the new congestion management system. That model identified a possible situation that could occur when the connection is fully (100%) utilized, which is an extreme congestion state. If a group of high consumption state cable-modems are operating at low priority (BE), and a new heavy cable-modem appears at high priority (PBE), it is theoretically possible that the high priority cable-modem with PBE QoS can take enough bandwidth away from the low priority BE QoS cable-modems that they appear to no longer be demanding heavy use. At the next measurement interval the low priority cable-modems are promoted to high priority (BE to PBE), and the just recently “new” high priority cable-modem is put at low priority (PBE to BE). The heavy cable-modems at high priority (PBE) now can consume all of the available bandwidth and the low priority cable-modem (BE) is starved, or prevented from getting any bandwidth, until the end of the next measurement period (approximately 15 minutes).

This problem is closely related to the oscillation problem described above. Since the large group of heavy use cable-modems operating at low priority will need to meet a lower threshold before being allowed to be promoted to high priority, this starvation scenario is much less likely to occur. Furthermore, the described scenario only occurs when the network is consistently used at or near 100% utilization, which based upon trials conducted by Comcast, is extremely, extremely rare.

During the trials we worked hard to find locations where utilization was very high, and still did not find a situation that would cause this starvation. We also tested this possible use case in our lab and were unable to produce starvation, possibly due to the exact manner in which CMTS packet schedulers actually work in practice, which could not be adequately simulated in a model. The likelihood of this starvation problem occurring in a real network is very low.

All of that being said, we expect to continue to fine-tune the system. We also had help in performing this analysis — from an independent consulting firm (NetForecast, as you noted, http://www.netforecast.com). They’ll be continuing on as we implement and into 2009, to periodically audit system behavior and make independent recommendations.

optionshiftk

I am very fortunate that my ISP(cablevision) is not capping my bandwidth. If I were a comacast subscriber I would be running to FIOS or nearly anything else that beats out comcast.

Jason Livingood

You can find this information in the FCC filing that was submitted today. There are three application servers in the system. One is an IPDR server that collects statistics, and that vendor is not yet announced. The second is a congestion management server, from Sandvine. The third is a QoS server (PacketCable Multimedia really) from Camiant.

Regards
Jason

Robb Topolski

I’m looking at the Comcast Network Management Policy site but I’m not seeing any information that wasn’t already posted before today. Do you have any better of a link to it? (NEVERMIND, I SEE THEM AT THE BOTTOM OF YOUR STORY — you’re ahead of Comcast.net, I think!)

Jason Livingood

> “Comcast answered all but our questions about who is providing the equipment to enable it’s efforts.”

In the FCC filing, in Attachment B – Future Practices, you can find this information on page 14 (which is page 15 in the PDF due to the cover sheet). The first application server will be an IPDR server, which will collect relevant cable
modem volume usage information from the CMTS, such as how many aggregate upstream or
downstream bytes a subscriber uses over a particular period of time.15 Comcast has not yet
chosen a vendor for the IPDR servers, but is in active negotiations with several vendors.
The second application server is the Sandvine Congestion Management Fairshare
(“CMF”) server, which will use Simple Network Management Protocol (“SNMP”) to measure
CMTS port utilization and detect when a port is in a Near Congestion State. When this happens,
the CMF server will then query the relevant IPDR data for a list of cable modems meeting the
criteria set forth above for being in an Extended High Consumption State.
If one or more users meet the criteria to be managed, then the CMF server will notify a
third application server, the PCMM application server developed by Camiant Technologies, as to
which users have been in an Extended High Consumption State and whose traffic should be
treated as BE. The PCMM servers are responsible for signaling a given CMTS to set the traffic
for specific cable modems with a BE QoS, and for tracking and managing the state of such
CMTS actions.

Robb Topolski

I’m looking at the Comcast Network Management Policy site but I’m not seeing any information that wasn’t already posted before today. Do you have any better of a link to it?

Comments are closed.