One of the things that publishers of all kinds love about online media is that it can be measured in a thousand different ways: every pageview and every click can be tracked, and the amount of time a user spends on the page can be quantified, as well as where they came from and where they went after they visited. But as Jonathan Stray notes in a recent post at the Nieman Journalism Lab, a much harder question is why we are measuring these things at all. Is it for the benefit of advertisers? Is it to prove that we are accomplishing something worthwhile for society as a whole? Before we can properly measure whether online journalism — of any kind — is effective or not, we have to answer the question: What is journalism for?
As we’ve described a number of times, the advent of the web and social tools like blogs and Twitter and Facebook has disrupted virtually every aspect of the media industry, from books to movies. And one of the hardest hit has been the traditional newspaper business, which finds itself competing against a growing number of online entities for both audience attention and advertising revenue. In response, some publishers such as the Newhouse family’s Advance Publications have been cutting back on printing newspapers in places like New Orleans, leaving that city without a printed paper for several days of the week — and causing an outpouring of criticism.
How do we measure the impact of a newspaper?
Much of the reaction to these moves — including an open letter from famous New Orleans residents criticizing Advance and responses from media analysts such as New York Times media writer David Carr — have focused on the idea that an online newspaper can’t possibly have as much of an impact on a city like New Orleans as a printed paper can. How do we know? The short answer is that we don’t, because there is no easy or reliable way to measure the impact that a printed newspaper has on the society around it, apart from pure sales. As Stray puts it:
“Evaluating the impact of journalism is a maddeningly difficult task. To begin with, there’s no single definition of what journalism is. It’s also very hard to track what happens to a story once it is released into the wild, and even harder to know for sure if any particular change was really caused by that story.”
Newspapers have always argued that they are important because the stories they cover result in changes to legislation or in corrupt companies being investigated, or have other social benefits such as highlighting health problems, but in virtually all cases the evidence is anecdotal at best. And as Stray notes, this problem doesn’t become any easier online — where everything can and is measured — because it isn’t clear what constitutes success for any given piece of journalism. Is it the number of readers it draws, or the “engagement” they produce (via things like comments or tweets)? Can you find it by looking at the number of links to that story from other sources over time, the way that Google’s Page Rank does?
There are some efforts to try and answer some of these questions, including a project from Aron Pilhofer — a developer and head of the data-journalism team at the New York Times — that involves a Knight-Mozilla News fellowship. It’s not clear from Pilhofer’s description what exactly the project plans to focus on: he says he hopes to come up with a framework and a methodology that will allow news entities to measure something approximating their social impact. But again, the problem is what to measure. As Pilhofer puts it:
“We are awash in metrics, and we have the ability to engage with readers at scale in ways that would have been impossible (or impossibly expensive) in an analog world. The problem now is figuring out which data to pay attention to and which to ignore.”
Is serving subscribers enough, or is there more to it?
Most of the efforts at measurement that publishers have made so far consist of tracking eyeballs and responses so that advertisers can be sure they are reaching the right audience, because when you rely on ads for the bulk of your revenue that is the most important factor in your survival. But what about newspapers like the New York Times and Financial Times, which are either close to or have already become primarily funded by reader subscriptions? Surveying those readers to determine whether they feel satisfied is one way to quantify your success — but is that enough? What about the broader public mandate that journalism is supposed to have?
One of the things that complicates this whole process is the fact that “journalism” is so poorly defined in the first place, as Stray and others have noted. What do we mean we say that word? In most cases, people seem to mean investigative or meaningful reporting on global events like the war in Afghanistan or the crimes committed by hedge funds and banks. But the reality is that those kinds of stories make up a tiny fraction of the journalism produced by major newspapers and other outlets — they are vastly overwhelmed by entertainment news, “service” journalism about things like how to file your taxes, gardening columns and so on.
As news developer Stijn Debrouwere pointed out recently, much of what traditional media companies are competing with for attention and ad revenue doesn’t even look like journalism: things like Reddit’s “Ask Me Anything” features, or Tumblr’s viral content, or BuzzFeed’s animated GIFs. Those services don’t worry about what the social impact of their content is — all they care about is clicks, because they don’t have a social mandate. Newspapers don’t have that luxury, but nor do they have an easy metric to demonstrate their success.
That is going to become a crucial issue as the disruption of the industry continues to increase: How does a mainly digital media entity determine whether it is having an impact or not — or having the kind of social impact that we have come to associate with journalism? And if it can’t answer that question, then why do we need it?