7 Comments

Summary:

Stock traders and hedge funds can’t predict the future yet, but they are doing their best to come as close as possible, and that involves crunching every bit of data they can get their hands on — up to and including that tweet you just posted.

minorityreport3x2

In the movie Minority Report, the government tried to predict future crimes by using a trio of “pre-cogs” who had visions of what was going to happen while lying in some kind of chemical bath. Stock traders don’t have that kind of setup yet, but they are doing everything they can to predict the future — and that involves crunching every bit of data they can get their hands on, up to and including that tweet you just posted. The New York Times recently described this phenomenon, and a hedge fund says it plans to launch a new fund that will trade stocks based in part on an analysis of market sentiment as defined by Twitter.

Stock traders have always looked at market-sentiment indicators, including some who study fashion trends or popular music to try to determine which way stock indexes are likely to go. It’s a little like science and a little like voodoo, as even some traders will admit. But the key is information — as much information as possible — and investment banks and hedge funds now have more of it than they could ever want, and the computing power to take advantage of it by crunching and analyzing it.

In a sense, they are doing the same thing Google does, but they are doing it in order to figure out which stocks to buy, not because they want to serve up related advertising. This is just part of the future of what we call “big data.”

The Times story describes how information services such as Dow Jones, Thomson Reuters and Bloomberg are trying to cater to this demand for more data by adding features to the terminals and software they sell to banks and investment houses, which tries to parse the sentiment of news stories, blog posts and even Twitter messages based on the use of common words and emoticons like the “smiley.” One portfolio manager at an equity fund says he feeds that data into his trading systems, and that such features give him “the ability to assimilate more information.”

Meanwhile, a hedge fund called Derwent Capital Markets says it will launch a new fund in February that will trade based in part on analysis of Twitter sentiment. This approach is built on research from the University of Manchester and Indiana University (PDF link) that showed how the number of emotional words on Twitter could be used to predict moves in the Dow Jones index. Researchers said they found that a change in emotions as expressed on Twitter would be followed by a move in the index between two and six days later, and that this method had greater than 87-percent accuracy.

The fund is apparently going to use other data in its analysis as well, but since the fund company has signed an exclusive deal with the researchers who published the sentiment paper, it sounds like tweets will play a major role in the trading. Will it work? Some are skeptical — including Reuters blogger Felix Salmon. My only question is: How long until someone starts setting up spam or bot accounts to try to game specific stocks? I’d give it about a week — if they don’t exist already. Welcome to the future.

Related GigaOM Pro content (sub req’d):

Post and thumbnail photo courtesy of DVDActive

  1. Seems interesting..and I think it would work. Couple of years back there was a talk at Defcon, which talked about trading stocks by analyzing spam emails.

    Share
  2. Doing this with non-stock market specific tweets is meaningless. People may be “not calm” or calm for many reasons, especially teens and colleges kids with BF and GF, drugs and other growing issues, and most don’t even buy stock. A coincidental correlation is a coincidental correlation.

    On the other hand, if you analyze stock-specific sentiment using natural language understanding and domain-specific knowledge, such as knowing the meaning of buying a call, selling a put, issuing dividends, targets and expectations, etc., the analysis and correlation will be more meaningful. That’s what mktsentiment.com does. We have found filtering rules that have proven to be very effective and can be a useful supplement to other technical indicators.

    Spams are not a problem because each source is weighted by a vector and the spams are weighed zero or close to zero. Echoes are detected and treated differently too.

    Share
  3. That ending could not have been more perfect and spot on. Great article.

    Share
    1. Thanks, David — glad you liked it.

      Share
  4. Good story & genuine inputs. I second your thoughts published here. There are enough and more watch dogs….pundits… in the internet who can predict the stock market fairly accurate,than what they did 15 years ago with the internet technology. Tred with care on the cyberhighway;)

    Share
  5. The key is using other data in conjunction with this information. Here is a list of potential factors that will make the case stronger for such efforts to make sense of the unstructured social media content – http://wp.me/p1gRAJ-1A

    Share
  6. Perhaps if data was analyzed by taking random samples the effect of bot accounts could be neutralized? At any rate I wouldn’t give up basic Technical Analysis to trade securities. This doesn’t seem like it would add any measurable value to a trading system.

    Share

Comments have been disabled for this post