Doubt Cast On Research Suggesting Equal Time For TV And Online

A Forrester Research report released Monday received a ton of attention for its suggestion that TV and internet usage in the U.S. had reached parity. Now that data is drawing some high-profile skeptics.

The problem is Forrester’s findings don’t remotely square with existing measurement on TV and internet usage. While the study found that in January and February of 2010 consumers reported spending 13 hours per week on both TV and internet, data from Nielsen and comScore (NSDQ: SCOR), arguably the most reliable sources for measurement of TV and internet usage, offer a markedly different picture.

ESPN plans to meet Wednesday with Forrester, which counts the sports juggernaut as a client, to share its concerns. “Our fundamental concern is that, in a very confusing media landscape, we’re trying to answer very important questions about the behaviors of consumers,” said Dave Coletti, vice president of digital media research and analytics at ESPN (NYSE: DIS). “It’s imperative that we answer questions with the right methods.”

In the first quarter of 2010, Nielsen clocked weekly usage at 38 hours and 44 minutes, nearly three times what Forrester found. Over the same time frame, comScore’s account of internet usage was 7 hours and 24 minutes, about half of what Forrester found.

Why these numbers are so divergent cuts to the heart of the difficulties ESPN has with this Forrester study. The Forrester numbers are entirely based on self-reporting, or what the 30,000 respondents to the survey say is their consumption habits. But that’s a subjective metric different from the kind of metered measurement Nielsen and comScore do. They may have their own well-documented faults, but are at least they’re objective.

But what’s more troubling to Glenn Enoch, vice president of integrated research at ESPN, is that in media-research circles, self-reporting is known to be notoriously slippery. “It’s something we’re generally careful about,” he said.

To wit: In a Video Consumer Mapping study conducted last year by Ball State University Center for Media Design that is widely regarded as a landmark piece of research, one of the key findings noted, “Serious caution needs to be applied in interpreting self-report data for media use. TV was substantially under-reported while online video and mobile video usage were over-reported.”

On Twitter, a few prominent self-appointed critics openly questioned Forrester’s findings. Gian Fulgoni, chairman of comScore, had a rather heated exchange with Forrester’s lead researcher on the report, Jacqueline Anderson, in which he questioned the validity not of his own stock in trade–Internet measurement–but the TV numbers.

“Nielsen says TV 140+hrs /mo. You say it’s only 52. Something very wrong,” he tweeted. After a few back-and-forth tweets, Anderson defended the work as “clear” when you examine the year-over-year numbers. “Clear?” Gulfoni shot back. “There’s huge error level.”

Reached for comment, Anderson doesn’t take issue with the veracity of Nielsen or comScore’s numbers. But she feels they are important pieces of a puzzle that isn’t complete without getting the consumers’ perspective. There’s the reality of what metered measurement yields, but to Anderson there is also value in distilling the perception of what consumers believe is the reality of their consumption habit. “In their minds’ eye now, the time consumers spend between mediums is equal,” said Anderson.

In her defense, Anderson stated clearly in a blog post about the research on Forrester’s site that “the data we present in this most recent Technographics® report is self-reported, so the metrics aren’t the same as those you’d see from a Nielsen or comScore.”

However, the very first line of the executive summary on the page where Forrester makes the full research available to its clients seems to pass off the research as what viewers actually consume instead of what they think they consume: “For the first time ever, the average US online consumer spends as much time online as he or she does watching TV offline.”

Anderson believes that the TV vs. internet comparison is not as significant in this research as the growth–or lack thereof–that each medium experienced since 2009. TV consumption didn’t decrease; it just held steady while internet made the huge leaps it took to catch up. “The data in the year-over-year picture is the more important piece of the puzzle,” said Anderson, who also noted that she was cognizant that respondents tend to under-report, but if they’re doing so year-over-year, it’s an apples-to-apples comparison.

But that nuance was apparently lost on the dozens of press outlets who wrote about the research, trumpeting it as some kind of milestone in the growth of U.S. internet usage yet failing to convey that self-reporting isn’t the best basis for declaring a tie between the mediums’ exposure levels. Few referenced the wealth of statistics that demonstrate just how largely TV consumption still looms over internet usage, a dynamic one industry researcher recently put in perspective by characterizing the Facebook audience as being on par with that of PBS. Maybe it’s hard to resist that sexy narrative of the underdog coming from behind to race neck-and-neck with the longtime leader.

Then there’s the very either-or premise of the research to consider. The distinction between TV and online usage isn’t even entirely clear anymore in a universe in which there are a bevy of boxes that deliver programming directly to the TV set via broadband connection. And concurrent usage of TV and online is already a well-noted phenomenon, making any presentation of data that paints TV vs. online as a zero-sum game off the mark.

Perhaps it’s predictable that ESPN would be the one to want to counter the study. Just last week, the network released its own study that sought to minimize the so-called cord-cutting phenomenon, which drew observations that ESPN was only trying to protect its gravy train. And of course, the Forrester research is now being touted as supporting evidence of cord-cutting.

“When we do raise an eyebrow at things like this is, it’s often interpreted as we’re downplaying the potential of digital media,” said Coletti. “Nothing can be further from the truth. I wave the flag for digital media. It’s more about putting on my professional researcher hat and making sure the data that gets into the marketplace is as accurate and reliable as could be.”