Blog Post

Some advice for media: Just because you can measure something doesn’t make it important

One of the best things about the web is that it provides almost an infinite number of ways to measure things, from what browser people use or how big their screen is to the amount of time they spend on a page. But this ability to measure can also be one of the worst things about the web, at least from a publishing point of view, as Knight-Mozilla OpenNews fellow Stijn Debrouwere pointed out in a recent post — because it encourages media companies and publishers of all kinds to focus on incremental increases in largely meaningless numbers, instead of paying attention to the things that actually matter.

In his post, which is taken from a presentation he made to a Hacks and Hackers meetup in Berlin earlier this month, Debrouwere — a member of the Guardian’s data team — compares the media obsession with analytics to a “cargo cult.” That’s the bizarre sociological phenomenon that took place in remote parts of the world after the World War II, in which islanders tried to bring back the gods (i.e., the U.S. military) by replicating their behavior:

“They’d put on uniforms and wave around makeshift signal cones, the ones you use to tell a plane where and how to land. Cultists would construct and man communications shacks and talk into radios with nobody at the end of the line. They had gotten a taste of modern technology, medicine and entertainment and they wanted more.”

Analytics don’t have magical properties


For many media companies, Debrouwere argues, data analytics have taken on this kind of role: since publishers know that technology companies use analytics and “big data” all the time to generate value, they assume that using the same kinds of tools — even if they don’t really understand what they are measuring or why — will produce the same kind of magical results, and thereby transform their businesses from money-losing dinosaurs into sleek and modern internet players. But in many cases, as Debrouwere puts it, they are talking into radios with nobody at the end of the line.

I like Debrouwere’s analogy, in part because I think the whole phenomenon of cargo cults is fascinating, but for me it misses the mark a little. Media companies aren’t trying to bring back something they already had by using analytics — it’s more like they were remote villagers hidden in the rain forest who had never seen a ruler or a scale for measuring weight before, and suddenly when the web came along they were handed these tools and didn’t really know what to do with them. So naturally, they ran around measuring the length and height and weight of everything in sight, without really knowing why.

To compound the problem, the whole advertising model for publishing online was (and still largely is) built on the idea of traffic — and more recently, metrics like unique visitors — and so those became the most important things to measure. And of course, since whatever you measure is also what you tend to produce, pageviews became the goal, which led inexorably to the kind of online media skewered by the Onion during the recent Miley Cyrus “twerking” scandal, not to mention slideshows and other tools. As DeBrouwere puts it:

“Pageviews is a vanity metric: something that looks really important but that we can’t act on and that tell us nothing about how well we’re actually doing, financially or otherwise.”

Real-time data compounds the problem


More recently, we’ve seen the rise of real-time analytics measurement thanks to tools like Chartbeat from Betaworks, which provides a dashboard of things you can measure — and even a “heads-up display” that shows you how many people are reading an article right now, how many have made it past a certain point, and other incredibly depressing metrics. But Debrouwere is right when he says that most publishers are not equipped to turn on a dime and change their output based on real-time analytics, and so they are often more of a frustration or a distraction.

To be fair to Chartbeat, it also provides a lot of useful things to track, such as the amount of time readers spend on a page and whether they have commented or not, as well as tracking who has been talking about or sharing your piece on social media — which is fascinating to watch. Unfortunately, many publishers and writers (and I am as guilty of this as anyone) just watch the big headline number that shows how many people are reading a specific page or post, because that’s what feels the most important. Says Debrouwere:

“There’s nothing like a dashboard full of data and graphs and trend lines to make us feel like grown ups. Like people who know what they’re doing. So even though we’re not getting any real use out of it, it’s addictive and we can’t stop doing it.”

So what is the solution? There isn’t an easy one. Debrouwere notes that the best way to use analytics is to (surprise!) actually have some idea of what you are trying to achieve and why it matters, and then use specific tools to see if that is actually happening — trying to promote more undiscovered content, for example, or encourage readers at a specific time of day. In other words, measure the length and weight of things for some larger purpose, not just because you can. Be sure to read Debrouwere’s whole post.

Post and thumbnail images courtesy of Shutterstock / Sarin Kunthong and Shutterstock / Nomad Soul

12 Responses to “Some advice for media: Just because you can measure something doesn’t make it important”

  1. Stephen Spark

    People like metrics because they give them the illusion of control. I worked for a company whose middle management had become completely obsessed with metrics – at times it felt as though we spent more time measuring what we were doing than actually doing it! The perceived need to record every aspect of what we were doing certainly made us less, rather than more, efficient, but junior and middle managers felt they had to have figures to prove to their bosses that they were fulfilling the (arbitrary and therefore meaningless) goals handed down to them. Of course, people soon got wise to the pointlessness of all this and just submitted any set of figures that looked vaguely convincing, so the whole edifice was built on sand. Stressing about the number of people who read only halfway down the page is the modern-day equivalent of debating how many angels can dance on a pinhead.

    Measurement/analytics/metrics should be used only for limited periods and for tightly defined purposes with a particular business-related end in mind. And if they are distracting people from getting on with their jobs, then they should be ditched.

  2. The thing about such analysis is, like statistics, they can be moved around and changed to fit in with what a company purchasing such information wants to hear and are still a goldmine for those producing ‘facts’ and figures. The industry still has a long way to go, even Facebook hasn’t managed to get the analysis right yet, and they have a captive audience providing all the information they require voluntarily.

  3. Just like any other major strategic business decision, analytics must be treated with proper planning, staffing, and management. Throwing money at an analytics system and expecting results without knowing what you’re doing is like throwing money at programmers and telling them to “make something useful,” largely a waste of time and money.

  4. Robb Montgomery

    I was there when Debrouwere presented this at Zeit Online in Berlin. Without a slide deck and only notes folded over in his hands. His presentation style was important.
    He was not showing, only telling.

    He indicated that publishers like analytics (and will buy any new analytics tool that surfaces) even if they don’t pay attention to the proper signals.

    These guys want big, blinking, impressive dashboards sitting on their desks so that they can feel like they are the captain of the ship.

    He also said indicated that publishers like the Guardian serve far too many stories each day as related to average consumption. To wit I asked him in the Q&A followup session. “So can Analytics guide us to making better decisions about which types of content and/services should go on Web vs. Tablet vs. smartphone . . .”

    He said “no, not really. Analytics tools were built for e-commerce – for salespeople.”

    I have to say he Cargo Cult story was fun, too.

  5. Good stuff and I say the same and have been for a few years now. When it comes to what you see in the media, great video by Charlie Siefe, mathematician at NYU, who wrote the book “Proofiness, the Dark Arts of Mathematical Deception”…I keep it as one of my best educational videos in the footer of my blog along with 3 other good ones..

    Every query and every stick of code does NOT have value like they want you to believe.

  6. Paul Barter

    Sounds a bit like the Homer Simpson advice to Bart – ‘If something is hard to do then it’s not worth doing’. Of course managers in ANY industry shouldn’t focus on meaningless metrics. The most important job of ANY manager is to decide WHAT to measure.