Size doesn't always matter

When it comes to media, not everything that counts can be counted

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

One of the great ironies of the online media business is that there are more ways to measure reader activity than there have probably ever been in the history of human communication — pageviews, unique visitors, time spent, clickthroughs, etc. — but no one can seem to agree on which measure accurately reflects the value that content creates. It’s like quantum mechanics: Our tools have never been better, but the thing we are trying to measure still slips from our grasp.

A sense of frustration at this state of affairs seeps out of a recent post by Evan Williams, the former co-founder and CEO of Twitter and founder of Medium, the site that is part content platform and part traditional publisher (something he also recently posted some thoughts about, after some criticism from Pando Daily’s Sarah Lacy). The industry still seems obsessed with large numbers such as pageviews or unique visitors, he says, but this results in sites and services that are “a mile wide and an inch deep.”

“We literally say one company or service is bigger based on a single number?—?specifically, number of people who have used it in the last 30 days. Even without even getting into how use is defined, this is dumb.”

Pictures vs. tweets

This is particularly obvious when comparing services like Twitter and Instagram, says Williams, returning to the scene of an earlier argument. As he put it in his original comment to Fortune, Twitter is “what we wanted it to be. It’s this realtime information network… important stuff breaks on Twitter and world leaders have conversations on Twitter. If that’s happening, I frankly don’t give a shit if Instagram has more people looking at pretty pictures.”

Instagram new filters

His point, Williams says, isn’t that Instagram is somehow less valuable because it involves pictures, but that the two services are trying to do completely different things, and so measuring them with the same blunt instrument — whether it’s called unique visitors or monthly average users — tells you virtually nothing of any value about their likelihood of success.

Even replacing those kinds of crude measurements with something like time spent probably doesn’t help with services like Twitter, Williams notes, since it is somewhat similar to Google, in that part of what it does is link to things, and thereby send people elsewhere. How does one measure the value of that?

“Twitter is as much utility as media and part of its benefit is being incredibly concise?—?plus, a lot of its purpose, like Google, is to point people off to other places. So optimizing for time spent probably wouldn’t make sense.”

When it comes to sites like Medium, however, looking at time spent gets closer to what they are trying to achieve, Williams says — so the site looks at a metric it calls “total time reading,” which is similar to what Chartbeat terms “total engaged time” and Upworthy refers to as “attention minutes.” These metrics are an attempt to discover real engaged users, as opposed to the drive-by click traffic that is never going to return.

Time spent vs. clicks

In Medium’s case, for example, a recent post that went viral on Facebook drove traffic numbers through the roof: “By number of unique visitors to, we blew it out of the park,” says Williams, “but the vast majority of those visitors stayed a fraction of what our average visitor stays, and they read hardly anything.” And so the company was much happier with a week in October when the unique visitor count was 40 percent lower, but time spent was 50 percent higher.


Although he touches on it somewhat — where he notes that Wall Street continues to value companies based on outdated metrics like monthly average user — the Medium founder doesn’t really grapple with the underlying cause of the media’s obsession with pageviews or unique visitors: namely, the fact that the advertising industry continues to focus on those raw numbers, regardless of whether they actually denote anything of value, because the ad industry is arguably even more backward than the media.

There are signs of hope here and there, with Chartbeat pushing the time-spent measurement and being accredited to do so for advertising (which will make the metric more appealing for both publishers and advertisers) and certain publications like the Financial Times and The Economist convincing ad agencies to go with time-spent measurement instead of raw visitors. But it’s going to take more than that to shift the industry towards measuring actual engagement rather than the equivalent of window-shopping or foot traffic.

The biggest problem is that there is no single “God metric,” as BuzzFeed founder Jonah Peretti put it in a recent interview — no over-arching measurement that both describes value and is easy for advertisers to understand. It changes depending on what the content is and what the goal is. As sociologist William Cameron put it: “Not everything that can be counted counts, and not everything that counts can be counted.” That may be hard for advertisers and publishers to accept, but it’s true.

7 Responses to “When it comes to media, not everything that counts can be counted”

  1. Retired Fraudster

    the real issue is how everyone hacks/manipulates data and metrics. click fraud is just one particularly egregious example. many of the app store charts are just overrun with fraud. as are “like” counts and twitter followers. as are quantcast and nielsen and compete and all ostensibly neutral third-party numbers

  2. Christopher Sessums

    The previous comments are dead on.
    The “not everything that counts can be counted” is a fallacious answer.
    We can easily build better instruments to collect data and measure how people use content or spend their time online.
    The truth is, real research takes more effort than folks will allow.
    Surveys and existing site data are only part of the equation.
    More insight can be gathered by interviewing hundreds of users over several iterations.
    So much can be learned by sitting down with users and observing them and asking them why they do what they do.
    This, of course, requires human capital, time, and money.
    And who wants to pay for this effort when companies and advertisers are so easily bamboozled?
    This level of focus and research will separate long standing companies from the short term ones over time.

    • Chris Stephenson

      Good points, but interviewing individuals not only isn’t practical, it’s probably not all that helpful either. I’ll guaranty people have no idea how many steps a day they take. They over-exaggerate how many times a week they go to the gym. They don’t know accurately how many ski runs they get in or how fast they ski, or what their highest altitude was. But if they are running the right apps on their phones (and their wrists), they will find out. Data is the answer, but we need to right systems in the place and we need to measure the right things. And, to your point, no one has yet taken the time and painstaking care to create these systems yet.

      • Christopher Sessums

        Point taken.
        I’ve conducted a couple of usability studies of how users engage with devices/content. You’d be amazed how much you can learn by watching people perform tasks. Interviews depend so much on the questions being asked and the demeanor of the interviewer. It’s a bit of an art mixed with science. Yet such attention to detail is ultimately what we’re all interested in providing whether it’s from the data the system collects to the data the researcher collects. I’m willing to bet if we can combine these forces (humans and systems) we’ll all be better off :)

        • Chris Stephenson

          It’s hard to argue with that :) – you certainly have the experience with these kinds of studies. It’s pretty interesting, that after all this time, advertisers and publishers alike are using fairly blunt instruments to gauge all this stuff. There’s a lot of room to improve and innovate in this space. It’ll be fun to swim in these choppy waters over the next few years.

  3. gsatell

    What you say is true, but also a bit of a red herring. In reality, advertisers decide what they want and then find the numbers to support their decision.

    The dirty little secret of the ad industry is that media buyers can’t count. Their math skills are atrocious!

  4. You don’t even define value here. Is it economic value ,is it social, before you can measure it, you need to define it.
    We are really bad at measuring things, how we define media is a minor problem.
    Look how we define the well being of country and where it gets us. The focus is on the economy and economic growth so politicians sell us out to global corporations.
    Measuring the economy is a very poor way of doing things and many other things need to be factored in. We do have all kinds of indicators but we don’t seem able to find a good enough solution..
    Guess simple and flawed wins because it’s easier to understand it.
    When it comes to Wall Street, you do have to factor in that they mostly care about short term financial results, so it’s all about the now. The medium and long term is factored in but to a lesser degree so the less complex ways of measuring things are better suited for them.