25 Comments

Summary:

An executive at a firm ISPs hire to audit their broadband meters says most of his clients so far haven’t built accurate meters.

For the 64 percent of Americans whose internet service provider imposes a broadband cap, and for those lucky enough to have a meter, I have some bad news. The president of the firm who audits many of the country’s broadband meters says that he can’t certify the measurements produced by five out of seven of his clients’ meters because they don’t count your bits correctly.

Peter Sevcik, president of NetForecast, told GigaOM that seven clients have hired his firm to audit their broadband meters over the last few years, but of those seven only one — Comcast — has published a report on the NetForecast certification. Sevcik is only willing to certify one other client in a public report.

Meters are a black box

The other five clients — which Sevcik would not name — have meters that Sevcik views as inaccurate, although not all of them have publicly rolled out their meters. And not all of those clients impose a broadband cap. Sevcik usually expects accuracy on the meters of between plus or minus one percent, but so far these don’t measure up.

“They are wrong by missing numbers by one way or another — sometimes it’s over reporting, but more frequently the error is under reporting,” he said. Under reporting should be a relief to those facing overage charges or service termination for going over their meters, but if the meters aren’t counting the data properly, it is still a problem.

Also disturbing is the attitude that Sevcik has encountered at some clients with malfunctioning meters. “There’s a general sense by some people, ‘Eh, we under report so we give them a free pass, so why worry about that?’” Sevcik says. “I think one does need to worry because it ruins the overall veracity of the meter. It derails trust in the meter.”

Broadband caps have grown to cover more Americans. They often come with meters.

Broadband caps have grown to cover more Americans. They often come with meters.

Sevcik wouldn’t name those clients, but his website lists Time Warner Cable, Cox, Comcast, AT&T, Bell Canada, Verizon and France Telecom as customers. Time Warner Cable and Cox have both confirmed to me that they have used NetForecast to certify their meters. While Verizon doesn’t have a broadband cap or a meter, Verizon emailed to confirm that it is not using NetForcesat on the wireline side. AT&T’s spokesman says it has a team of engineers that certifies the accuracy of its meters but that it hasn’t worked with NetForecast to certify its wireline meters. Sevcik clarified that the seven clients he’s speaking of are all U.S.-based and all are testing wireline meters.

Last November, AT&T customer Ken Stox drew attention to AT&T’s meters when he couldn’t replicate the ISP’s byte count with his own home testing. For Stox, who is technically astute, the questions he had about the meter were less about fairness and more about understanding what, when and how AT&T was counting.

Building a broadband meter is tough

pitydafoolThose same questions are ones that Sevcik hopes ISPs will answer as part of an overall effort to improve their meters. He notes that whatever you think about the fairness of data caps, if meters are to serve some kind of public purpose, the public has to understand what the ISPs are counting and how they are counting it.

As for problems that lead to inaccurate meters, there are several. The first is that many of these meters are bolt-on afterthoughts. A telco or a cable company often uses measurement gear that sits on the subscriber side of the network. The ISPs has to allocate enough resources at that point to track the bits properly, but networks become congested. Then the ISP faces a choice. Does it count all the bits and risk slowing down the network, or does it let the bit count slide and let the rush of packets through?

Most ISPs err on the side of letting them rush through and a better user experience. But to solve the problem they could dedicate more resources to the counters so they can keep up with peak traffic. More resources would also solve the next problem ISPs face — once they have the bit counts, they need to add them up. As Sevcik describes it, many of these counters drop the bits into an Internet Protocol Detail Report format. Those reports are generated every 15 minutes.

Spread that across 10 million subscribers with a goal of doing hourly updates, and suddenly you have 40 million records to process in that hour. That takes servers — in some cases more than the ISP anticipated.

And while Sevcik said that while some ISPs had used decimal counting as opposed to binary counting of bytes in the past, most used binary counting today. That’s good because a binary count adds about 7 percent to the total number of bytes. But as Sevcik points out, if a consumer streams 3 HD movies on Saturday night and expects to see that jump in data consumed on his usage meter, then it needs to be there, or the consumer needs to know why.

Are meters worth it?

It’s clear that building a meter takes work. A Time Warner Cable spokesman notes that development of its meter took several years — other ISPs said it took at least a year of effort from multiple engineering teams.

My broadband consumption courtesy of Time Warner Cable. Not sure how I consumed 44GB in only 6 days.

My broadband consumption courtesy of Time Warner Cable.

If building a meter is so much work and consumes so many resources, why have them? For example, Comcast, which delayed the rollout of its meters while it struggled to get it right (and still runs monthly accuracy checks) has defended its meter as a customer education tool and as a means to manage network consumption. Critics point out that at 300 GB, Comcast’s cap is suspiciously close to the 288 GB figure that Comcast has said would be the amount of data consumed by someone using their broadband to replace cable. Those critics generally call caps a way for ISPs to protect their pay TV businesses.

What we do know is that Comcast has spent a lot of money and effort making sure its meter is accurate, because as Charlie Douglas, a Comcast spokesman notes, “We knew it would be in the spotlight.” I imagine also because it knew a meter would be the first step in how it could change the pricing dynamic from all-you-can-eat to something that’s a little bit more metered. And as I have pointed out in previous articles, if meters become the basis for charging subscribers overage fees or even terminating their service, then someone needs to monitor those meters to ensure that they are accurate.

I’ve called on the FCC to wake up and start gathering more data on how meters affect consumers and whether or not they are accurate, but the agency has so far been content to let this experiment in caps pay out without much oversight. With these accusations maybe the FCC will finally step up. Clearly, as a country we’re moving toward capped and metered broadband.

Sevcik, whose experience goes back to the days of the ARPANET and the first routing systems, believes if that’s the case, then those meters should be accurate.

“I’ve been in the internet business for quite some time … and in that time I’ve had my hand in the design of more than 100 networks and seen a lot in network technology. And what I’ve realized is, as the industry has matured there is an awful lot of talk and decisions made by people — consumers, policy advocates in DC and big companies — that is often based on hype,” he said. “my goal in a small way in this world of hype is to shed a light of real data and make a little piece of it really right.”

In short; If we’re going to accept meters on our broadband, then let’s make sure they are accurate.

This story was updated at 11:49 a.m. PT with comment from Verizon.

  1. Yeah, we warned about that nearly 4 years ago…

    http://www.zatznotfunny.com/2009-05/heres-why-you-want-bandwidth-caps/

    The elephant in the room that no one wants to talk about is the utility provider cannot be the one who provides the monitoring tools – it is a conflict of interest. Your home has a physical meter that measures your consumption of electricity. The device is constructed by third parties, independently verified accurate by a Federal or state agency and, like a gasoline pump, is also surrounded by multiple laws governing the behavior of the the provider, protecting the Consumer from fraud. If bandwidth caps and metered billing are a “foregone conclusion” then objective, independent monitoring of consumption is too.

    Share
    1. Damn straight, Dave. My hope is that there will be more and more credible people stepping up to show how inconsistent and opaque these meters are, and someone (the FCC? Sen. Wyden?) will step up.

      I certainly won’t stop writing about it :)

      Share
      1. I’m kinda wondering what NetForecast’s goal or agenda with this is… what’s in it for them to pretty much badmouth their clients? Is it a misguided attempt to get rehired for additional auditing or to engineer more accurate solutions? Hm.

        Share
      2. And, yeah, the common people need more, stronger government representation. Seems like the providers call most of the shots, free of oversight.

        Share
  2. ISPs don’t care because meters are meant to scare customers to use less than what they actually pay for. ISPs want people to believe in the myth that bits are scarce to justify their high prices.

    Share
  3. And this is why I installed a router with Tomato Firmware to independently monitor my bandwidth. I wanted to know how much I was using, and to have evidence in case I was accused of exceeding my bandwidth cap.

    Share
    1. That’s great Greg, but do you know how our ISPS actually tracks usage? Ken Stox, who was mentioned in this story was doing the same thing, and was having trouble reconciling his data with AT&T’s and AT&T didn’t help.

      Share
  4. I’m curious how this applies to companies like FreedomPop. I know they’ve tried to hit me with extra charges for supposedly going over my limit due to a 3 hour delay in their usage numbers.

    Share
  5. As usual, GoogleOm PR rep Stacey Higginbotham lobbies for Google’s agenda: no caps, no limits, no profits for ISPs so that advertiser Google can make all of the money in the Internet ecosystem.

    Share
    1. Is that you, Brett? Let’s be fair here; there are plenty of normal customers on the Internet who will ask to get something for nothing. IMO there’s no need to invoke Google shilling.

      Share
    2. If what you call “Google’s agenda” is best for consumers and the development of broadband, is it still a problem?

      Look at the advances in broadband speed and access in other countries, the cost of that service, and maybe you’ll see that our current large ISP monopolies we have here aren’t quite the poor struggling unsung heroes you think them to be.

      Share
  6. This story is strictly related to wireline meters as opposed to how wireless firms track data, so I’m not sure how FreedomPop tracks it’s usage. Wireless companies have over the years had to adapt the way the charge in relation to consumer pressure.

    Share
    1. Your headline is wrong Todd. The meters are in use, they just aren’t being used to charge customers or enforce caps. I noted in the story that some of these meters are not being used to enforce caps, but some are in use.

      Share
      1. Sevcik told me none of those five ISPs has launched bandwidth meters.

        Share
      2. Here is a headline that even *better* clarifies the issue:

        About Those Faulty Broadband Meters: None of Them Are Used to Track Usage Limits
        http://www.multichannel.com/blogs/bit-rate/about-those-faulty-broadband-meters-none-them-are-used-track-usage-limits

        Share
    2. Todd, I rely on my meter from TWC to understand how much bandwidth I consume. It informs my stories and how I think about broadband. TWC doesn’t have a cap, nor does it charge me per byte. BUT if its meter is wrong I still think that’s a problem. I’m drawing conclusions and advocating for policies based on that data.

      So you can say this isn’t an issue, but I think facts are always worth getting right. In meters and in your reporting.

      Share
      1. Time Warner Cable says NetForecast has certified its usage meter.

        Share
      2. (Which I reported in my post yesterday.)

        Share
  7. America is so behind on this! Look at the Netherlands, nobody has caps here. The ISP’s can easily supply you with whatever amount of usage you need.

    The reason they have caps is to reduce pirating, they don’t want people to upload stuff.
    Are you guys really still going to jump through all those hoops and spend all those resources so publishers can keep oppressing people?

    I buy all my games and it does not bother me that some people download them for free.

    The reason we have piracy is that publishers will abuse their position to make us pay for the same crappy content ten times over. Piracy forces them to deliver quality service and take the customer serious.

    Share
  8. Good article… but something not mentioned is that every byte is being counted by the TCP/IP stack… all that is needed to measure the absolute number of bytes is to regularly query that number from the stack and keep a tally in non-volatile storage.

    However, customer data should always be considered as only data successfully transmitted (don’t count retries and corrupt data… it’s the ISP’s fault the data is being lost) but not data that the customer hasn’t requested (like malware probes and misdirected packets).

    What that all boils down to is that the meter MUST be on the customer’s end, and behind a Stateful Packet Inspection firewall as with Greg’s Tomato firmware (good firmware, BTW).

    Share
  9. I’ve had AT&T Uverse for several years. So far, their meter has been useless…

    http://www.johnharlow.com/UverseMeter.png

    I especially like the part where they tell me not to worry.

    Share

Comments have been disabled for this post