3 Comments

Summary:

According to new research by Twitter’s data science team, Twitter search is used often as a tool for finding breaking news in real time, which makes it difficult for Twitter to assign relevance to any given tweet or topic in the long run.

sleeping_grid-miguel

Is it possible that Twitter’s users, rather than Twitter itself, are to blame for the micro-blogging platform’s relatively useless search engine? Perhaps. According to new research by Twitter’s data science team, Twitter search is used often as a tool for finding breaking news in real time, which makes it difficult for Twitter to assign relevance to any given tweet or topic in the long run. So while the world bemoans Twitter search as useless, maybe we’re doing so through last generation’s Google-colored glasses that don’t let us see Twitter for what it is and the challenges it faces.

In a Twitter Engineering blog post explaining its findings, analytics research scientist Jimmy Lin explains the problem of ranking tweets by relevance as partly being a problem of time. In the case of breaking news, the system is simply overwhelmed by tweets and queries on that topic, which means Twitter’s relevancy models can’t always keep up to determine which ones you probably want to see. While it’s relatively easy to build a simple search algorithm utilizing the concept of “term frequency-inverse document frequency weight” when the overall corpus of documents is fairly static, it’s a lot harder when terms suddenly surge in popularity and a system has to constantly re-process the dataset in real time.

These numbers from Twitter’s research help explain the problem:

  • Examining all search queries from October 2011, we see that, on average, about 17% of the top 1000 query terms from one hour are no longer in the top 1000 during the next hour. In other words, 17% of the top 1000 query terms “churn over” on an hourly basis.
  • Repeating this at a granularity of days instead of hours, we still find that about 13% of the top 1000 query terms from one day are no longer in the top 1000 during the next day.
  • During major events, the frequency of queries spike dramatically. For example, on October 5, immediately following news of the death of Apple co-founder and CEO Steve Jobs, the query “steve jobs” spiked from a negligible fraction of query volume to 15% of the query stream — almost one in six of all queries issued! Check it out: the query volume is literally off the charts! Notice that related queries such as “apple” and “stay foolish” spiked as well.

Of course, this particular phenomenon doesn’t explain why Twitter’s search doesn’t go back further in time, or why its algorithms for ranking tweets based on source or the number of time they’ve been retweeted don’t appear too accurate. Even if relevancy improves, there’s still a lot to be desired in terms of getting Twitter to return the types of results users have come to expect.

Lin’s post also highlights another piece of research from Twitter that’s less noteworthy to individual users but probably more telling about the world as a whole. A visualization of Twitter usage patterns in New York City, Tokyo, Sao Paulo and Istanbul creates a picture of cultural and seasonal differences at play.

Twitter users in Tokyo, we see, tweet a lot less during the work day and also go to bed and wake up at about the same times throughout the year. Elsewhere, users show pretty distinct differences in activity as the seasons change. Lin also points out the afternoon lull in Sao Paulo. It’s difficult to discern the exact reason from looking at this chart, but the lull does coincide with Sao Paulo’s winter season and a generally later beginning to the tweeting day.

I’d love to see these results analyzed against other cultural datasets, or even just against a knowledge base of local customs and behaviors, to see how Twitter  use — and web use, generally — comports or doesn’t comport with a region’s typical norms.

You’re subscribed! If you like, you can update your settings

  1. Frank Speiser Monday, June 4, 2012

    This actually makes perfect sense, and Lin is correct in his research. For instance, you already know quite a bit about what people are interested in based on selection bias and what accounts they tend to follow. That tells you what they’re likely to “mean” and what they prefer to see.

    The results, in order to be accurate need to be bounded by time and by what you’re likely to care about given the signal in things your’re likely to be connected to. It takes a lot of computing to do this, but that’s what it takes to be accurate.

    It also makes intuitive sense when you think about it this way: “What do you want to talk about this time tomorrow?” I would hope the answer is that you probably don’t know, and therefore you need to compare all of the things you could talk about, that you know about, that you have an interest in and then as humans, we’ll do an approximate ordinal calculation of what is most engaging. Those are the tweets you’ll want to see.

    The people in the broader see if Tweets in cases like this are actually noise in a lot of cases. Users that tend to follow Account A may be inclined to prefer a topic much more than users of Account B, especially given the current conversations they are seeing and have access to. It’s complicated, but the results of this study are not surprising.

    It’s why we’re able to be so effective here at SocialFlow. We’ve put quite a bit of effort into figuring this out, and it’s a very difficult problem to solve. You need to have the right mindset to even frame the problem, and you have to be willing to learn through multiple game-turns – because there is no way, as a human, you’ll be able to predict the number of things that can vary and impact people’s preference for discussion. Life is simply too complex for that, but it’s also what makes it cool to be alive, and free to engage in sharing thoughts.

    Great article.

  2. Gilad Mishne Monday, June 4, 2012

    The purpose of this study was to demonstrate one particular challenge of searching Tweets, and how we at Twitter are investing substantially in innovative solutions. For example, we make significant usage of social and interest interactions with tweets as part of our ranking, and personalize each search; the unique data we have access to allows us to perform well even where traditional models such as tf-idf fail.

    Millions of Twitter users use our search product every day, with a high rate of returning visits and interactions on the search results, and we are hard at work to make their experience better every day.

    1. Gilad,
      Thanks for the comment and for the research, which is really quite interesting. While the headline is a bit provocative, I don’t think it’s inaccurate to say that many users aren’t too fond of Twitter’s search feature. I think this research (and, I hope, my post) explain one reason why that that is, and I have no doubt Twitter is always striving to make search better.

Comments have been disabled for this post