More bad news about broadband caps: Many meters are inaccurate

25 Comments

For the 64 percent of Americans whose internet service provider imposes a broadband cap, and for those lucky enough to have a meter, I have some bad news. The president of the firm who audits many of the country’s broadband meters says that he can’t certify the measurements produced by five out of seven of his clients’ meters because they don’t count your bits correctly.

Peter Sevcik, president of NetForecast, told GigaOM that seven clients have hired his firm to audit their broadband meters over the last few years, but of those seven only one — Comcast — has published a report on the NetForecast certification. Sevcik is only willing to certify one other client in a public report.

Meters are a black box

The other five clients — which Sevcik would not name — have meters that Sevcik views as inaccurate, although not all of them have publicly rolled out their meters. And not all of those clients impose a broadband cap. Sevcik usually expects accuracy on the meters of between plus or minus one percent, but so far these don’t measure up.

“They are wrong by missing numbers by one way or another — sometimes it’s over reporting, but more frequently the error is under reporting,” he said. Under reporting should be a relief to those facing overage charges or service termination for going over their meters, but if the meters aren’t counting the data properly, it is still a problem.

Also disturbing is the attitude that Sevcik has encountered at some clients with malfunctioning meters. “There’s a general sense by some people, ‘Eh, we under report so we give them a free pass, so why worry about that?'” Sevcik says. “I think one does need to worry because it ruins the overall veracity of the meter. It derails trust in the meter.”

Broadband caps have grown to cover more Americans. They often come with meters.

Broadband caps have grown to cover more Americans. They often come with meters.

Sevcik wouldn’t name those clients, but his website lists Time Warner Cable (s twc), Cox, Comcast (s cmcsa), AT&T (s T), Bell Canada, Verizon (s vz) and France Telecom as customers. Time Warner Cable and Cox have both confirmed to me that they have used NetForecast to certify their meters. While Verizon doesn’t have a broadband cap or a meter, Verizon emailed to confirm that it is not using NetForcesat on the wireline side. AT&T’s spokesman says it has a team of engineers that certifies the accuracy of its meters but that it hasn’t worked with NetForecast to certify its wireline meters. Sevcik clarified that the seven clients he’s speaking of are all U.S.-based and all are testing wireline meters.

Last November, AT&T customer Ken Stox drew attention to AT&T’s meters when he couldn’t replicate the ISP’s byte count with his own home testing. For Stox, who is technically astute, the questions he had about the meter were less about fairness and more about understanding what, when and how AT&T was counting.

Building a broadband meter is tough

pitydafoolThose same questions are ones that Sevcik hopes ISPs will answer as part of an overall effort to improve their meters. He notes that whatever you think about the fairness of data caps, if meters are to serve some kind of public purpose, the public has to understand what the ISPs are counting and how they are counting it.

As for problems that lead to inaccurate meters, there are several. The first is that many of these meters are bolt-on afterthoughts. A telco or a cable company often uses measurement gear that sits on the subscriber side of the network. The ISPs has to allocate enough resources at that point to track the bits properly, but networks become congested. Then the ISP faces a choice. Does it count all the bits and risk slowing down the network, or does it let the bit count slide and let the rush of packets through?

Most ISPs err on the side of letting them rush through and a better user experience. But to solve the problem they could dedicate more resources to the counters so they can keep up with peak traffic. More resources would also solve the next problem ISPs face — once they have the bit counts, they need to add them up. As Sevcik describes it, many of these counters drop the bits into an Internet Protocol Detail Report format. Those reports are generated every 15 minutes.

Spread that across 10 million subscribers with a goal of doing hourly updates, and suddenly you have 40 million records to process in that hour. That takes servers — in some cases more than the ISP anticipated.

And while Sevcik said that while some ISPs had used decimal counting as opposed to binary counting of bytes in the past, most used binary counting today. That’s good because a binary count adds about 7 percent to the total number of bytes. But as Sevcik points out, if a consumer streams 3 HD movies on Saturday night and expects to see that jump in data consumed on his usage meter, then it needs to be there, or the consumer needs to know why.

Are meters worth it?

It’s clear that building a meter takes work. A Time Warner Cable spokesman notes that development of its meter took several years — other ISPs said it took at least a year of effort from multiple engineering teams.

My broadband consumption courtesy of Time Warner Cable. Not sure how I consumed 44GB in only 6 days.

My broadband consumption courtesy of Time Warner Cable.

If building a meter is so much work and consumes so many resources, why have them? For example, Comcast, which delayed the rollout of its meters while it struggled to get it right (and still runs monthly accuracy checks) has defended its meter as a customer education tool and as a means to manage network consumption. Critics point out that at 300 GB, Comcast’s cap is suspiciously close to the 288 GB figure that Comcast has said would be the amount of data consumed by someone using their broadband to replace cable. Those critics generally call caps a way for ISPs to protect their pay TV businesses.

What we do know is that Comcast has spent a lot of money and effort making sure its meter is accurate, because as Charlie Douglas, a Comcast spokesman notes, “We knew it would be in the spotlight.” I imagine also because it knew a meter would be the first step in how it could change the pricing dynamic from all-you-can-eat to something that’s a little bit more metered. And as I have pointed out in previous articles, if meters become the basis for charging subscribers overage fees or even terminating their service, then someone needs to monitor those meters to ensure that they are accurate.

I’ve called on the FCC to wake up and start gathering more data on how meters affect consumers and whether or not they are accurate, but the agency has so far been content to let this experiment in caps pay out without much oversight. With these accusations maybe the FCC will finally step up. Clearly, as a country we’re moving toward capped and metered broadband.

Sevcik, whose experience goes back to the days of the ARPANET and the first routing systems, believes if that’s the case, then those meters should be accurate.

“I’ve been in the internet business for quite some time … and in that time I’ve had my hand in the design of more than 100 networks and seen a lot in network technology. And what I’ve realized is, as the industry has matured there is an awful lot of talk and decisions made by people — consumers, policy advocates in DC and big companies — that is often based on hype,” he said. “my goal in a small way in this world of hype is to shed a light of real data and make a little piece of it really right.”

In short; If we’re going to accept meters on our broadband, then let’s make sure they are accurate.

This story was updated at 11:49 a.m. PT with comment from Verizon.

25 Comments

Steve Cotton

I posted my first item before reviewing all the other comments, but since I also oversee TM Forum’s Customer Experience Management area, I’d observe that the accuracy and precision of meters for broadband is a central issue for CEM. Trust is at the core of successful CEM, and that starts with the simple relationship of one’s bill to one’s perception of usage.

While enterprise customers can probably afford (and should do) independent monitoring/auditing of their usage, that might not be practical for consumers. The best defense, where an open market exists, is to rely upon that mechanism to drive people to the trustworthy provider. No amount of intervention by government will do anything but raise cost and lower quality (as always).

Steve Cotton

While I currently serve as Director, Business Assurance at TM Forum and thus oversee the Forum’s IPDR work, I was employee #1 in IPDR.org as well, and also worked with Larry Roberts and Barry Wessler (ARPANET pioneers) in GTE Telenet to ensure that the meters used for the X.25 packet networks were “billing grade”. On top of that, this problem is hardly new or unique–the timing of disconnect signals in the trunk network of wireline carriers, sometimes off by a single second per call, was audited by systems AT&T and GTE put in and often accounted for millions of dollars of settlement corrections.

A real challenge for the current environment is that the broadband connections are “always on”, with many bits of traffic transiting both ways, unbeknownst to the customer (for example-when an automatic update to Windows occurs in the middle of the night, I’d venture to say that the customer hasn’t made the connection that this traffic is controlled by their settings in Windows, and does contribute to their usage). One way to help this would be if the IPDR traffic would be segregated by customer usage versus performance monitoring (that latter is where the terabyte a year sort of figures creep in). That would also allow the meters to more accurately reflect customer-visible usage.

A tradeoff that was done for economical reasons in the old days was a store-and-forward node at the switch, but that flys in the face of both the expectations of the customers to have near real-time meters tracking their usage, as well as the resources at the remote end that have to be dedicated to making that a practicality.

This never has been a simple problem, so I wouldn’t expect it to get easier in the internet age.

FilmRic

I find it ludicrous that they using caps to protect their cable TV business. I dont reach my cap and watch a ton of content and will never go back to cable content until the idiots gett off their high horse and offer ala carte. If they dont go down that road they are going to be relegated to dumb pipe status whether they want to be or not.

Its time to compete cable!

elfonblog

Good article… but something not mentioned is that every byte is being counted by the TCP/IP stack… all that is needed to measure the absolute number of bytes is to regularly query that number from the stack and keep a tally in non-volatile storage.

However, customer data should always be considered as only data successfully transmitted (don’t count retries and corrupt data… it’s the ISP’s fault the data is being lost) but not data that the customer hasn’t requested (like malware probes and misdirected packets).

What that all boils down to is that the meter MUST be on the customer’s end, and behind a Stateful Packet Inspection firewall as with Greg’s Tomato firmware (good firmware, BTW).

Free citizen

America is so behind on this! Look at the Netherlands, nobody has caps here. The ISP’s can easily supply you with whatever amount of usage you need.

The reason they have caps is to reduce pirating, they don’t want people to upload stuff.
Are you guys really still going to jump through all those hoops and spend all those resources so publishers can keep oppressing people?

I buy all my games and it does not bother me that some people download them for free.

The reason we have piracy is that publishers will abuse their position to make us pay for the same crappy content ten times over. Piracy forces them to deliver quality service and take the customer serious.

Stacey Higginbotham

Your headline is wrong Todd. The meters are in use, they just aren’t being used to charge customers or enforce caps. I noted in the story that some of these meters are not being used to enforce caps, but some are in use.

Stacey Higginbotham

Todd, I rely on my meter from TWC to understand how much bandwidth I consume. It informs my stories and how I think about broadband. TWC doesn’t have a cap, nor does it charge me per byte. BUT if its meter is wrong I still think that’s a problem. I’m drawing conclusions and advocating for policies based on that data.

So you can say this isn’t an issue, but I think facts are always worth getting right. In meters and in your reporting.

Stacey Higginbotham

This story is strictly related to wireline meters as opposed to how wireless firms track data, so I’m not sure how FreedomPop tracks it’s usage. Wireless companies have over the years had to adapt the way the charge in relation to consumer pressure.

B Glass

As usual, GoogleOm PR rep Stacey Higginbotham lobbies for Google’s agenda: no caps, no limits, no profits for ISPs so that advertiser Google can make all of the money in the Internet ecosystem.

Wes Felter

Is that you, Brett? Let’s be fair here; there are plenty of normal customers on the Internet who will ask to get something for nothing. IMO there’s no need to invoke Google shilling.

mz

If what you call “Google’s agenda” is best for consumers and the development of broadband, is it still a problem?

Look at the advances in broadband speed and access in other countries, the cost of that service, and maybe you’ll see that our current large ISP monopolies we have here aren’t quite the poor struggling unsung heroes you think them to be.

Austin Johnsen

I’m curious how this applies to companies like FreedomPop. I know they’ve tried to hit me with extra charges for supposedly going over my limit due to a 3 hour delay in their usage numbers.

Greg

And this is why I installed a router with Tomato Firmware to independently monitor my bandwidth. I wanted to know how much I was using, and to have evidence in case I was accused of exceeding my bandwidth cap.

Stacey Higginbotham

That’s great Greg, but do you know how our ISPS actually tracks usage? Ken Stox, who was mentioned in this story was doing the same thing, and was having trouble reconciling his data with AT&T’s and AT&T didn’t help.

Virtuous

ISPs don’t care because meters are meant to scare customers to use less than what they actually pay for. ISPs want people to believe in the myth that bits are scarce to justify their high prices.

Dave Zatz

Yeah, we warned about that nearly 4 years ago…

http://www.zatznotfunny.com/2009-05/heres-why-you-want-bandwidth-caps/

The elephant in the room that no one wants to talk about is the utility provider cannot be the one who provides the monitoring tools – it is a conflict of interest. Your home has a physical meter that measures your consumption of electricity. The device is constructed by third parties, independently verified accurate by a Federal or state agency and, like a gasoline pump, is also surrounded by multiple laws governing the behavior of the the provider, protecting the Consumer from fraud. If bandwidth caps and metered billing are a “foregone conclusion” then objective, independent monitoring of consumption is too.

Stacey Higginbotham

Damn straight, Dave. My hope is that there will be more and more credible people stepping up to show how inconsistent and opaque these meters are, and someone (the FCC? Sen. Wyden?) will step up.

I certainly won’t stop writing about it :)

Dave Zatz

I’m kinda wondering what NetForecast’s goal or agenda with this is… what’s in it for them to pretty much badmouth their clients? Is it a misguided attempt to get rehired for additional auditing or to engineer more accurate solutions? Hm.

Dave Zatz

And, yeah, the common people need more, stronger government representation. Seems like the providers call most of the shots, free of oversight.

Comments are closed.