Peter Sevcik, president of NetForecast, told GigaOM that seven clients have hired his firm to audit their broadband meters over the last few years, but of those seven only one — Comcast — has published a report on the NetForecast certification. Sevcik is only willing to certify one other client in a public report.

Meters are a black box

The other five clients — which Sevcik would not name — have meters that Sevcik views as inaccurate, although not all of them have publicly rolled out their meters. And not all of those clients impose a broadband cap. Sevcik usually expects accuracy on the meters of between plus or minus one percent, but so far these don’t measure up.

“They are wrong by missing numbers by one way or another — sometimes it’s over reporting, but more frequently the error is under reporting,” he said. Under reporting should be a relief to those facing overage charges or service termination for going over their meters, but if the meters aren’t counting the data properly, it is still a problem.

Also disturbing is the attitude that Sevcik has encountered at some clients with malfunctioning meters. “There’s a general sense by some people, ‘Eh, we under report so we give them a free pass, so why worry about that?’” Sevcik says. “I think one does need to worry because it ruins the overall veracity of the meter. It derails trust in the meter.”

Broadband caps have grown to cover more Americans. They often come with meters.

Sevcik wouldn’t name those clients, but his website lists Time Warner Cable, Cox, Comcast, AT&T, Bell Canada, Verizon and France Telecom as customers. Time Warner Cable and Cox have both confirmed to me that they have used NetForecast to certify their meters. While Verizon doesn’t have a broadband cap or a meter, Verizon emailed to confirm that it is not using NetForcesat on the wireline side. AT&T’s spokesman says it has a team of engineers that certifies the accuracy of its meters but that it hasn’t worked with NetForecast to certify its wireline meters. Sevcik clarified that the seven clients he’s speaking of are all U.S.-based and all are testing wireline meters.

Last November, AT&T customer Ken Stox drew attention to AT&T’s meters when he couldn’t replicate the ISP’s byte count with his own home testing. For Stox, who is technically astute, the questions he had about the meter were less about fairness and more about understanding what, when and how AT&T was counting.

Building a broadband meter is tough

Those same questions are ones that Sevcik hopes ISPs will answer as part of an overall effort to improve their meters. He notes that whatever you think about the fairness of data caps, if meters are to serve some kind of public purpose, the public has to understand what the ISPs are counting and how they are counting it.

As for problems that lead to inaccurate meters, there are several. The first is that many of these meters are bolt-on afterthoughts. A telco or a cable company often uses measurement gear that sits on the subscriber side of the network. The ISPs has to allocate enough resources at that point to track the bits properly, but networks become congested. Then the ISP faces a choice. Does it count all the bits and risk slowing down the network, or does it let the bit count slide and let the rush of packets through?

Most ISPs err on the side of letting them rush through and a better user experience. But to solve the problem they could dedicate more resources to the counters so they can keep up with peak traffic. More resources would also solve the next problem ISPs face — once they have the bit counts, they need to add them up. As Sevcik describes it, many of these counters drop the bits into an Internet Protocol Detail Report format. Those reports are generated every 15 minutes.

Spread that across 10 million subscribers with a goal of doing hourly updates, and suddenly you have 40 million records to process in that hour. That takes servers — in some cases more than the ISP anticipated.

And while Sevcik said that while some ISPs had used decimal counting as opposed to binary counting of bytes in the past, most used binary counting today. That’s good because a binary count adds about 7 percent to the total number of bytes. But as Sevcik points out, if a consumer streams 3 HD movies on Saturday night and expects to see that jump in data consumed on his usage meter, then it needs to be there, or the consumer needs to know why.

Are meters worth it?

It’s clear that building a meter takes work. A Time Warner Cable spokesman notes that development of its meter took several years — other ISPs said it took at least a year of effort from multiple engineering teams.

My broadband consumption courtesy of Time Warner Cable.

If building a meter is so much work and consumes so many resources, why have them? For example, Comcast, which delayed the rollout of its meters while it struggled to get it right (and still runs monthly accuracy checks) has defended its meter as a customer education tool and as a means to manage network consumption. Critics point out that at 300 GB, Comcast’s cap is suspiciously close to the 288 GB figure that Comcast has said would be the amount of data consumed by someone using their broadband to replace cable. Those critics generally call caps a way for ISPs to protect their pay TV businesses.

What we do know is that Comcast has spent a lot of money and effort making sure its meter is accurate, because as Charlie Douglas, a Comcast spokesman notes, “We knew it would be in the spotlight.” I imagine also because it knew a meter would be the first step in how it could change the pricing dynamic from all-you-can-eat to something that’s a little bit more metered. And as I have pointed out in previous articles, if meters become the basis for charging subscribers overage fees or even terminating their service, then someone needs to monitor those meters to ensure that they are accurate.

I’ve called on the FCC to wake up and start gathering more data on how meters affect consumers and whether or not they are accurate, but the agency has so far been content to let this experiment in caps pay out without much oversight. With these accusations maybe the FCC will finally step up. Clearly, as a country we’re moving toward capped and metered broadband.

Sevcik, whose experience goes back to the days of the ARPANET and the first routing systems, believes if that’s the case, then those meters should be accurate.

“I’ve been in the internet business for quite some time … and in that time I’ve had my hand in the design of more than 100 networks and seen a lot in network technology. And what I’ve realized is, as the industry has matured there is an awful lot of talk and decisions made by people — consumers, policy advocates in DC and big companies — that is often based on hype,” he said. “my goal in a small way in this world of hype is to shed a light of real data and make a little piece of it really right.”

In short; If we’re going to accept meters on our broadband, then let’s make sure they are accurate.

This story was updated at 11:49 a.m. PT with comment from Verizon.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What the shift to the cloud means for the future EPG
• Who and what to watch in the new era of the living room
• A look back at mobile in Q1

What we talk about when we talk about caps.

The rise in caps has let ISPs influence the internet in subtle ways — most of which seem harmful to innovation. The first is to take away the idea that wireline broadband is an unlimited service, despite the ability of smaller ISPs to build out networks that don’t come equipped with caps. As you can see from the chart below, most of the ISPs are implementing overage charges associated with their caps. This isn’t really about managing their networks for congestion. If it were, they’d implement a different type of pricing model that cost users more to surf at peak times. No, this is about protecting their entrenched TV businesses as well as keeping the price for service high, despite the decreasing costs to send traffic over the network.

It’s also about grabbing more of the profits from the growth in internet services such as Netflix and Google, although caps take out those frustrations on users as opposed to the over-the-top providers. Instead of providing faster speeds for users and encouraging the growth of services that would require users to upgrade to those speeds, ISPs have taken their control of the last mile and are charging for bytes. So instead of paying more for better service, customers will pay more for what they use. This is a model that works for certain industries (think gasoline and electricity) but when it comes to encouraging more usage and innovation on the internet, the utility model seems short-sighted. Other ISPs may be thinking this same way.

For example, what if Intel had told game developers or Microsoft not to write software that would stress its chips — or penalized programmers for every megahertz of performance they used over a certain threshold? We’d end up with crappy software running on slower machines. Instead Intel encouraged people to write software for its chips and invested billions in making them faster so people would upgrade. Along the way it opened up market after market for the PC. Utility industries aren’t typically hotbeds of innovation.

The Federal Communications Commission, which is charged with tracking the spread and quality of U.S. broadband, has so far been quiet on this issue, not even collecting data to track how the shift to capped broadband has affected users, much less the industry. That may be changing. But it’s time that we ask if we want the internet to look like the utility or a source of continued innovation.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What the shift to the cloud means for the future EPG
• Who and what to watch in the new era of the living room
• A near-term outlook for big data

A spokesman for the FCC notes that the FCC is still looking at both the spectrum and the competitive issues associated with the deal. Those competitive elements are the marketing agreements and the Joint Operating Entities that the cable companies and Verizon would form, could mean for consumers.

As I’ve written, I think both would be bad, with the Joint Operating Entities being potentially worse for consumers as the cable providers and Verizon seek to control technology that they could use to cut other players out of consumer’s homes — to the consumer’s detriment. As I wrote in February:

The marketing agreements create a shadow joint-operating entity (JOE) between Verizon and the cable companies. This JOE is worrisome to those of us who realize that getting Verizon in a room once a month with the executives at the nation’s largest cable companies could lead to agreements about technology, deployment strategies and R&D that will be controlled by the large ISPs. The fear is that this organization will be able to slowly stifle new innovations for Internet services or even devices attached to wireline networks by creating technologies and standards that are only available to the JOE participants. Perhaps others might be able to license those technologies, but there’s no guarantee of that, or that the JOE would do so for a fair and reasonable amount.

Verizon’s expanding its fiber-to-the-home empire has goaded the cable companies to constantly boost their broadband speeds, but if the deal get approved that competitive dynamic disappears. It also means cable companies might not sign partnerships with smaller, or hungrier wireless companies such as T-Mobile or Sprint.

So far, the idea is that the DoJ and/or the FCC could put some conditions on the JOE to prevent collusion among the cable companies and Verizon as well as try to extract concessions related to guaranteed speed improvements, but those conditions would then require dogged enforcement and only last so long.

If there is any doubt in the regulators’ minds that this deal is primarily about airwaves, they should put the kibosh on the marketing and joint operating entity associated with the deal and watch the cable companies renege on the deal. Nationwide licenses with 20 to 30 MHz of spectrum for $4 billion is pretty cheap. My hunch is if the marketing agreements and JOE were killed, we’d see the cable companies try to back out and get the true value of those airwaves.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from mobile’s second quarter
• 12 tech leaders’ resolutions for 2012
• Carrier IQ and the continued erosion of operator trust

Most of the new roaming functionality will be in New York City and the surrounding Tri-State area, Los Angeles, Tampa, Orlando, and Philadelphia as those areas have overlapping cable and Wi-Fi coverage. The group will be adding more cities and coverage areas, however. Signing in to a Wi-Fi network out of your home coverage area should be seamless: You’ll simply use the same credentials as you would on your own cable operator’s network.

As I’ve mentioned in the past, not only will consumers be happy with this broader Wi-Fi coverage, but so too will the mobile broadband network providers: Free Wi-Fi — even from someone else’s hotspot network — helps to offload data from 3G and 4G networks. Single sign-on while roaming on different networks makes this an easy win for consumers while cable providers can use the expanded coverage as a marketing perk to attract new consumers.

Sounds like a win all around to me and to my colleague Stacey, who said this two years ago after speaking to the folks at BelAir Networks, maker of wireless solutions, and now owned by Ericsson:

“So will ISPs take the consumer love of ubiquitous broadband and carriers’ need for offload to the next level and create the equivalent of roaming agreements for Wi-Fi? Greg Williams, the new SVP of corporate development at Bel Air Networks, thinks they might. …He wonders if carriers will negotiate with each other and fixed-line ISPs to get access for their wireless subscribers, especially in congested cities such as New York or San Francisco.”

Two years later, and it looks like the Wi-Fi roaming revolution is well underway. Be sure to bring your smartphone or tablet with you to get on board!

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Confused about the wireless markets? Here’s a breakdown
• The future of Wi-Fi in the enterprise
• Why retailers should forget showrooming and turn to in-store Wi-Fi

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The future of Wi-Fi in the enterprise
• ESPN Leads the Way Over the Top, But Will Others Follow?
• Connected consumer first-quarter 2013: Analysis and outlook

“Cox is working to make this transition as seamless and easy as possible for our customers,” EVP of Product Development Len Barlik said in the statement. “We are proud of our employees’ dedication to delivering the excellent customer service that Cox is known for, and we will continue to keep our wireless customers’ satisfaction a top priority during this transition period.”

Cox originally planned to become a full-fledged wireless operator, buying advanced wireless service (AWS) and 700 MHz at auction in the cities where it offers cable service. It tapped Chinese vendor Huawei to build a CDMA network, partnered with Sprint to provide roaming access outside of its territory and even hinted at a future LTE build to add mobile broadband to its already extensive home and business broadband service portfolio.

However, Cox got off to slow start, launching it’s “Unbelievably Fair” service in just a handful of markets in late 2010: a year behind schedule. Cox added more markets in 2011, but it showed signs it was having second thoughts about its mobile strategy. Rather than use Huawei network gear, Cox turned to its roaming partner Sprint to power its service while it expanded into new cities, until May when it revealed that it planned to junk its networks entirely but continue to provide service as a Sprint mobile virtual network operator (MVNO).

Now, with wireless available in roughly half of its cable territory, Cox is abandoning mobile entirely. Why did it fail? With the waves of consolidation in the wireless market, it’s becoming increasingly more difficult for a small wireless operator to make it on its own. The money Cox would sink into its own infrastructure wouldn’t produce much in the way of returns. While remaining an MVNO would allow it to maintain its quad-play bundles, the math probably stopped making sense as demand for home phones wanes. Cox was probably merely replacing its customers’ cable telephony service with a far less profitable mobile phone.

The big question is what Cox will do with its spectrum. Cox has already said it plans to sell off whatever networks it has built, but it will find few buyers. A CDMA network using Huawei gear at AWS would be incompatible with all of the major operators networks. The spectrum licenses, however, would be valuable to almost every operator. Leap Wireless and MetroPCS could use the spectrum to grow their regional CDMA and LTE footprints. Verizon Wireless and AT&T both are using 700 MHz for LTE and plan to expand into the AWS bands. T-Mobile’s high-speed packet access plus (HSPA+) network is built at AWS. Sprint is probably the only operator with no conceivable interest in the spectrum. If a bidding war hasn’t started already, Cox is definitely going to see a lot of suitors.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Where new opportunity lies in the mobile operating system space
• New solutions for the evolving mobile network
• Mobile 2012 and beyond

Sprint CEO says no to the AT&T and T-Mo merger.

Cox Communications says it plans to stop building out its wireless network according to Fierce Wireless, which published a story this afternoon citing Cox spokesman David Deliman who said the cable company will “soon” decommission its 3G network. The story noted that Cox, a cable provider, would outsource the wireless operations to Sprint.

So what happens to Cox’s AWS spectrum it is using to deploy its wireless network? Given that AT&T is planing to buy T-Mobile in part because it wants to deploy its 4G LTE network in that spectrum band, I imagine Cox determined that it might better off selling some airwaves and partnering up with Sprint.

This would give Sprint yet another customer to provide 3G access for, with others being Clearwire and Comcast. Even LightSquared, the promised satellite network, could end up sharing resources and wireless towers with Sprint. This puts Sprint in a contradictory position. There are multiple companies relying on it to provide 3G coverage for their 4G or even 3G networks, while its CEO is testifying before the Senate that it will likely cease to exist if AT&T succeeds in buying T-Mobile, the nation’s fourth largest carrier. The AT&T buy would cement AT&T and Verizon as the largest mobile operators and leave Sprint with a mere 18 percent of the nation’s subscribers.

However, as the only viable alternative to the top two players, Sprint could see its fortunes rise as other companies trying to force their way into the mobile broadband game. Plus, as it announces its 4G plans (and yes, it’s going to have to move to LTE especially since Clearwire is planning its move to LTE) it may become a key roaming partner for other operators. Sprint did once tell me it wanted to become a true wholesaler back when it was optimistic about its partnership with Clearwire, so maybe it’s dreams may finally come true as consolidation eats away at competition in the industry.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sprint’s tightrope walk: finding a balance for its network modernization plan
• Mobile 2012 and beyond
• Mobile Q4: The scramble for spectrum continues

The issue became apparent for the first time last week when Suddenlink said that it would love to offer its customers Netflix access on its Premiere DVRs, but can’t because Netflix isn’t able to deliver its subscription service to DVRs leased by cable companies. Netflix at the time confirmed that its contracts simply don’t allow this, and now Hulu is saying that it is bound to the same restrictions: “Currently, Hulu Plus will not be available on MVPD-provided TiVo Premiere DVRs,” we were told by a company spokesperson.

This type of blackout for online subscription services is frustrating for consumers who expect TiVo’s supplied by their cable company to offer the same type of service a TiVo sold through retail channels would get them. Suddenlink, which rolled out TiVo availability in four Texas communities this month, charges end users $15 per month to lease a Premiere DVR. That’s only $5 less than the no-money-down promotional offer recently introduced by TiVo.

However, the real loser could be TiVo itself: The company has been banking on alliances with cable companies to make up for its ever-declining consumer subscriber base. Its access to extra content has been one of the selling points. In fact, the company boasts on its website that it “brings the operator content, navigation, and branding options that they can’t get elsewhere,” with one of those content options mentioned being Netflix. Cable operators are unlikely to invest a lot of money into rolling out Premiere DVRs if those boxes offer little more than the generic devices they’ve been shipping to customers for years.

A TiVo spokesperson didn’t provide any further details when contacted for this story, and instead said that it was “important to remember that TiVo offers a millions of pieces of broadband delivered content from providers beyond just Netflix, as well as integrated cable video on demand.”

Related content on GigaOM Pro: (subscription required)

• Three Reasons Hulu Plus is No Threat to Netflix
• Cord-cutting? Hold the Phone
• Pay-TV’s Ala Carte Tipping Point

The Plus Package, which includes whole-home DVR, more DVR storage and additional high-definition channels, will be available for a small additional fee to Cox subscribers. The package will be rolled out in select Cox markets by the end of the second quarter, and to the rest of Cox’s footprint by the end of the year.

The Trio guide was designed by Cox and Frog Design and implemented by NDS, and is aimed at making content discovery easier. The guide has three panels for browsing — one which includes channel listings, one which has a list of programming on those channels, and a third that and show information. In addition to the default viewing option, the programming guide also enables subscribers to view content across different content categories and genres.

The main attraction, though, is the way the program guide combines content from broadcast, on demand and time-shifted recordings, enabling users to find all content across those platforms in a unified search interface. With advanced metadata, users can also find related content for programming that they enjoy, based on genre, actor or related series. The new system allows multiple users to create individual profiles, which enables them to have personalized DVR and favorited search or show listings. In future releases, Cox plans to extend the program guide to also include a recommendation function, based on previous programming that users have watched or searched for.

While enabling personalized content recommendations, the data that Cox collects could also enable more interactive and targeted advertising through the new service. In a phone interview, David Grabert, media relations director for Cox, said that while the cableco was “very cognizant of privacy concerns,” the service would allow it to work with Canoe Ventures to provide smarter, more interactive and more engaging advertising through its advanced set-top boxes.

Cox is just one company working on improving its customer experience through an integrated and more easily searchable programming guide. In many ways, Cox’s unified search and discovery mechanism, as well as the ability to find related content by actor and genre, match similar capabilities that TiVo recently added to its latest DVRs.

Related content on GigaOM Pro:

Why Cable Operators Need an Apps Strategy (subscription required)