Blog Post

BuzzFeed’s deal with Facebook to measure political sentiment has one major flaw

If you spend any time on either Twitter or Facebook, it quickly becomes obvious that both are hugely popular forums for political discussions — not to mention flame wars, trolling and other forms of behavior. Since social platforms have become such a key component of political activity, BuzzFeed recently announced that it has formed a partnership with Facebook, one that will give it exclusive access to the social network’s “sentiment analysis” data. But such an experiment comes with some significant risks and caveats attached.

In a post about the new project, BuzzFeed editor-in-chief Ben Smith said that he believes Facebook — for better or worse — is either in the process of replacing or may have already replaced television as the battleground on which elections are won or lost in the United States, and this could have a huge impact on the way that campaigns are structured and even on the way that issues are discussed.

[blockquote person=”” attribution=””]”The vast new network of some 185 million Americans opens the possibility, for instance, of a congressional candidate gaining traction without the expense of television, and of an inexpensive new viral populism. The way people share will shape the outcome of the presidential election… a rawly powerful video may reach far more voters in a few hours than a multimillion-dollar ad buy; and it will reach them from trusted sources — their friends — not via suspect, one-way channels.”[/blockquote]

Sentiment analysis

According to Smith, the partnership with Facebook (which also includes ABC News), will give BuzzFeed “a powerful new window into the largest political conversation in America.” Using the data from Facebook’s billion-plus users, Smith says, reporters for the site will be able to ask Facebook for insight into how voters in Iowa feel about Hillary Clinton, or which Republican candidate is most popular with women.

Facebook sentiment data

As the BuzzFeed editor points out, sentiment analysis — especially for political purposes — is a field that is well known for being flawed, in part because human behavior is difficult to predict, but also because the tools for measuring sentiment are not accurate enough. Smith says that “if anyone can pull this off, it will be Facebook,” because it has huge amounts of data and smart data scientists on staff.

[blockquote person=”” attribution=””]”It has access to a far, far larger sample of natural language than any other social network. What’s more, that carries with it contextual data that can serve as a point of departure for sentiment analysis — the field, in particular, that allows people to include how they’re feeling or what they’re doing when they post status updates. And third, Facebook quite simply has some of the best data scientists in the world, and has built a company on a deep and comprehensive understanding of user data.”[/blockquote]

There’s no question that Facebook has massive quantities of data about human behavior, and the ability to parse that data at almost unbelievable levels of granularity. That’s why sociologists and other researchers often jump at the chance to do research like the “emotional manipulation” study that generated so much controversy earlier this year, when Facebook changed the appearance of users’ newsfeeds in order to see if that would change their behavior.

Facebook vs. the real world

At the same time, however, the human behavior that Facebook sees — the liking of certain content, the hiding of other content, as well as sharing and commenting and other forms of interaction — is a tiny window into what users are really thinking (although admittedly it’s probably better than phone surveys). And it can be distorted by a wide range of other factors, including the social ties between users, ties that might convince someone to “like” a status update even though they might disagree with it on a number of levels.

facebook data dislike generic

But for me, the biggest potential flaw in the BuzzFeed partnership is that the universe of behavior and sentiment that Facebook users are exposed to through the site is determined by Facebook’s algorithm — in other words, the sentiment that BuzzFeed is trying to track is being influenced by the site itself, in thousands of tiny ways that may be difficult or even impossible to detect.

We saw a concrete example of this phenomenon and its potential downside during the upheaval in Ferguson, Mo. after the shooting of a young black man by a police officer. Twitter was filled with debate about the incident, but Facebook users mostly saw videos of friends doing the “ice bucket challenge.” Imagine that same phenomenon, amplified a hundred times, taking place during an election.

Only Facebook knows for sure

As sociologist Zeynep Tufekci pointed out at the time, Facebook’s ability to influence what users see has a huge potential impact on social issues, including those involved in elections. In a sense, Facebook has taken over the role that a newspaper editor or TV producer had — but the factors behind its ranking of different topics is even less well understood, and its influence is arguably much larger than any single newspaper would ever have been.

Facebook has already shown that what it displays in a user’s newsfeed can have an impact on things like their voting behavior, as Micah Sifry described in a recent Mother Jones piece: close to 2 million people had their newsfeeds changed leading up to the 2012 elections, and voter turnout increased measurably.

What this means in practice is that any attempt to draw conclusions about specific areas of sentiment around political activity will be clouded by BuzzFeed’s inability to know exactly what the newsfeeds of those users showed them, or how that might have changed as a result of Facebook’s algorithm tweaks — unless BuzzFeed is getting far more detail about how the algorithm works than anyone else has gotten to date.

Post and thumbnail images courtesy of Thinkstock / Wavebreak Media

One Response to “BuzzFeed’s deal with Facebook to measure political sentiment has one major flaw”

  1. zahrgnosis

    Welcome to the Observer Effect, which is long documented. While you’re absolutely right that we have no way of knowing what Facebook’s algorithm will do to the outcome, that’s true of pretty much every observational technique, even, as you mention, phone surveys. I’m interested in your example of the Ferguson vs. Ice Bucket comparison, however. Any number of affects other than just Facebook’s algorithm could have been in effect. The Ice Bucket challenge was far more social, and Ferguson was far more news oriented, and there’s an existing bias in how people use the two services that could contribute to the contrast, regardless of which algorithms pushed content to the end users.

    Also while you’re correct it will affect the results in “thousands of tiny ways that may be difficult or even impossible to detect”, it also affects the true sentiment that you’re measuring (not just the proxy), in similar ways, and any algorithm, including none at all, have similar affects… someone who only reads an algorithmic-ally structured news feed in the evenings may be more likely to see a topic from the morning than if they read an unfiltered in-order feed. There’s always uncertainty in these measurements, but the vast amount of data floating around any of these networks should prove informative, even if only to inform us how not to use them; one simply has to be careful when performing the analysis — as always.

    There’s no perfect polling mechanism, and there are many flaws in every approach. But something is better than nothing, and a variety approaches can only help inform… this should be a good learning experience for all involved; I certainly want to give it a chance.