2 Comments

Summary:

Twitter may be an ever-flowing stream of information, but as it becomes a more mainstream source of news and commentary it also becomes a huge reservoir of data that can be analyzed, and that’s what startups like ThinkUp and DataSift and Gnip are trying to do

The idea of a stream of constantly changing real-time information wasn’t invented by Twitter, but it’s probably one of the closest approximations of that idea for many users. To some extent, the appeal is the same as any stream: to watch it flow by, to step into it now and then. But there’s also a lot of potential value in that stream if you could capture it and analyze it, and that ‘s where a number of startups and services are focusing their gaze, including ThinkUp — which just launched a 1.0 version — and DataSift, which also just launched. And Gnip plans to sell data from the Twitter firehose to analysts and traders who want to tap into the zeitgeist of the Twitter-sphere.

ThinkUp is an app designed by former Lifehacker editor Gina Trapani, and comes from Expert Labs, the non-profit technology incubator and advisory group run by Anil Dash that has helped the Obama administration (among others) figure out how to use digital technology for social purposes (Dash also runs a consulting firm called Activate media). When installed by an individual user or company, it takes in data from Twitter accounts — as well as Facebook pages and Google+ accounts — then makes it easy to see patterns or to analyze the effectiveness of certain tweets or updates.

ThinkUp tracks and analyzes, but also archives

I’ve been using the app for several months now, and it’s quite fascinating to see the amount of data available. (You can see some of my data here if you are interested.) The dashboard shows the most retweeted content, the times of day when tweets get the most activity, the individual users who respond most, and provides all kinds of charts and graphs for presenting that information visually. For me it’s mostly curiosity, but if you’re running a corporate account, I can see how this kind of data would be hugely useful (the White House has been using it for some time) — and you can export and even embed it easily, since ThinkUp has an open API.

Although the information and insights are important, both Dash and Trapani also mentioned another potential benefit of ThinkUp: It acts as an ongoing archive of all your content, whether it’s tweets or retweets or status updates or even photos and videos. As Dash points out, with any content you upload to a social network of some kind, there’s a very real risk it will one day disappear. And while Facebook and Google let you download your data, Twitter doesn’t provide any way of getting tweets you might have posted in the past. The company’s search only goes back about a week, and even services like Topsy are less than perfect if you want to find a specific message. Says Dash:

The companies behind these networks can, and someday will, destroy all of those moments. Delete them from the record. Forever. With no advance notice… history shows us that it happens. Over and over and over. The clips uploaded to Google Videos, the sites published to Geocities, the entire relationships that began and ended on Friendster: They’re all gone.

In addition to the archiving aspect, however, the really powerful feature that services like ThinkUp can offer is the ability to spot patterns in your Twitter or other social-networking activity. Looking at this data over time could help identify all kinds of different things, particularly for corporate users — what content produced a response and when, who the influencers with respect to your content are. (In what seems to be a veiled reference to Klout’s ranking of users, Dash also said ThinkUp “does not, and will never display some arbitrary score to your profile.”)

DataSift and Gnip also filter and resell the firehose

That’s also the idea behind more ambitious data-filtering services such as DataSift, which was started by British programmer Nick Halstead, which got access to the Twitter firehose as part of a deal with Twitter when it took over the retweet button he created as part of his startup Tweetmeme. DataSift uses semantic filtering to detect patterns in the Twitter stream and is aiming its service at corporations and brands that want to see what kind of reaction their content is getting in social networks (DataSift currently filters just Twitter, but plans to add Facebook and Google+). DataSift launched on Thursday, and is backed by GRP Partners and IA Ventures.

The other company that has access to the full Twitter firehose of data is Gnip, which signed a deal with Twitter to resell that data to companies last year. And Gnip on Thursday announced a partnership with StockTwits (see disclosure below) to provide a feed of both Twitter data and related information from the StockTwits investment community to traders and venture funds who are trying to find patterns in that social activity around stocks or companies. At least one hedge fund has an entire trading strategy that is based on an analysis of Twitter and sentiment related to stocks and the markets.

Not that long ago, Twitter was seen as mostly a social plaything, but as it has become a crucial player in everything from the revolutions of the Arab Spring to the planning and scheduling of television events like the Oscars, it has become more obvious that there is value in that stream, provided you have the right filters.

Disclosure: StockTwits is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

Post and thumbnail photos courtesy of Flickr user Luc Legay

You’re subscribed! If you like, you can update your settings

  1. Interesting article but I still need to start seeing some more real world applications of this data. Yes, its valuable data and there’s lots of it but where are the tools that actually solve my problems? Do like the direction that http://flum.es seems to be heading in this respect

    1. I’m inclined to agree with you John Day.

      I glanced at http://flum.es after reading your comment. Not bad at all! Curation is a huge problem, as all this data is meaningless unless there is a way to extract more value from it, one way or another, than I’ve seen evidence of to-date. There are higher-end, for-fee services e.g. Research.ly, Omniture’s social media analytics and of course Data Sift. Maybe they do something quite useful and impressive.

      Otherwise, as far as curation, that I’ve seen to date, and this is sad but true, http://twitter.com/!#/PopURLs was the most useful yet, for general purpose filtering and aggregation. Of course, that was without any custom tailoring.

  2. Marco Castellana Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/vhGeCVnV

  3. “#Twitter is a #stream, but it’s also a reservoir” http://t.co/Z6pv2opI @gigaom

  4. A good perspective on Twitter – “Stream, but also a reservoir” http://t.co/G2LNI17W #Twitter #data #social

  5. Wessel van Rensburg Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/DesjP6b5

  6. Twitter is a stream, but it’s also a reservoir http://t.co/5bGOB7bv ll RT @jaycbee

  7. RT @AnnTran_: Twitter is a stream, but it’s also a reservoir http://t.co/5bGOB7bv ll RT @jaycbee

  8. Twitter is a stream, but it’s also a reservoir http://t.co/ZXjTmQ8H ll RT @jaycbee @AnnTran_

  9. Christo Oosthuizen Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/69ync3VA | RT @twitt_erfolg_de

  10. Dr John Weiner Friday, November 18, 2011

    I think of it as The Fountain of Youth RT @DrVes: Twitter is a stream, but it’s also a reservoir http://t.co/WI9ElBYh

  11. Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  12. RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  13. “@mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/L7eblQkK” fascinating

  14. Sally Cachecho Friday, November 18, 2011

    #mktg250d Twitter is a stream, but it’s also a reservoir http://t.co/juDLRFNR

  15. Terri D Harings Friday, November 18, 2011

    RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  16. Greta Lundgaard Friday, November 18, 2011

    RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  17. RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  18. Twitter is a stream, but it’s also a reservoir – Track and Analyze Your Activity with @thinkupapp http://t.co/jru9LvJj via @gigaom

  19. Interesting Twitter insight: Twitter is a stream, but it’s also a reservoir – http://t.co/bwE6iCI2 via @gigaom

  20. Joseph McDonald Friday, November 18, 2011

    RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  21. Twitter Is A Stream, But It’s Also A Reservoir http://t.co/8Ue88jUz (via @GigaOM)

  22. “@AttentionUSA: Twitter Is A Stream, But It’s Also A Reservoir http://t.co/WYqv1WCj (via @GigaOM)” #socialmedia

  23. Twitter is a stream, but it’s also a reservoir http://t.co/3adqtod9

  24. Butterfly Effect Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir | #twittersphere #askTwitter – http://t.co/NQhsZiDO

  25. Chris McLaughlin Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/1DAXG57R #SocialMedia

  26. Patatinis Developers Friday, November 18, 2011

    RT @mclaughlinchris: Twitter is a stream, but it’s also a reservoir http://t.co/1DAXG57R #SocialMedia

  27. And it can be a stagnant backwater with quicksand border! MT @mclaughlinchris #SocialMedia is a stream AND a reservoir http://t.co/nts2Ldw8

  28. Mohamed Abdel Samad Friday, November 18, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/n2yfJHv2 via @zite

  29. Twitter is a stream, but it’s also a reservoir http://t.co/QSZrKM70

  30. Clarissa Womack Friday, November 18, 2011

    RT @mpesce: Twitter is a stream, but it’s also a reservoir http://t.co/PWkisi40

  31. http://t.co/mpz3rDDX Twitter is a stream, but it’s also a reservoir – GigaOm

  32. #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  33. Hadel S. Ma’ayeh ✓ Friday, November 18, 2011

    RT @2morrowknight #Twitter is a stream, but it’s also a reservoir http://t.co/bRhcGxQ9 /@mathewi @Gigaom *great read!*

  34. Twitter is a stream, but it’s also a reservoir, and there is value in mining and filtering it. Same for FB & G+: http://t.co/PxHZZxLT

  35. John Paul Larrivey Saturday, November 19, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/Uxl9SiEQ

  36. Twitter is a stream, but it’s also a reservoir: 17, 2011, 3:07pm PT 1 Comment The idea of a stream of constantly… http://t.co/to6qQfvg

  37. RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/p1NVuduO /@mathewi @Gigaom *great read!*

  38. Twitter is a stream, but it’s also a reservoir – http://t.co/uFhdTBm9 http://t.co/KKEdJqMf #twitter

  39. Twitter is a stream, but it’s also a reservoir – http://t.co/YE43EAoh: http://t.co/ZKtD8JCi

  40. RT @gigaom: Twitter is a stream, but it’s also a reservoir http://t.co/iY4URdu1

  41. Frank  Gaijin Hero Saturday, November 19, 2011

    RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  42. #Twitter is a stream, but it’s also a reservoir http://t.co/ZXo7dG2S /@mathewi @Gigaom *great read!* via @2morrowknight @silverstar22b

  43. Bebe Ƹ̵̡Ӝ̵̨̄Ʒ Sarah Saturday, November 19, 2011

    RT @2morrowknight #Twitter is a stream, but it’s also a reservoir http://t.co/z7ixMPVR /@mathewi @Gigaom *great read!*

  44. RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  45. RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  46. RAAK Social Media Saturday, November 19, 2011

    Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/AXPl8DxX

  47. Twitter is a stream, but it’s also a reservoir http://t.co/LEF4gWwF

  48. Twitter is a stream, but it’s also a reservoir – GigaOm http://t.co/j8zAcsMg

  49. Social Media LF Saturday, November 19, 2011

    Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/te5yQD1f

  50. Lori Shemek PhD, CNC Saturday, November 19, 2011

    Excellent! RT @2morrowknight #Twitter is a stream, but it’s also a reservoir http://t.co/ssL1GHHJ /@mathewi @Gigaom *great read!*

  51. “@AnnTran_: Twitter is a stream, but it’s also a reservoir http://t.co/0gm3aoxj ll RT @jaycbee”

  52. Twitter is a stream, but it’s also a reservoir – http://t.co/nCJExCsQ via @Gigaom RT @2morrowknight

  53. RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  54. Steve Korchynsky Saturday, November 19, 2011

    Twitter = constant flow of info, leaving yesterday’s tweets in the dust. Startups record and analyze ALL the data: http://t.co/koDFiMQS

  55. #Twitter is a stream, but it’s also a reservoir http://t.co/rAFSm26c /@mathewi @Gigaom *great read!* @2morrowknight

  56. Twitter is a stream, but it’s also a reservoir http://t.co/MTnxo3Aj

  57. http://t.co/j8zAcsMg Twitter is a stream, but it’s also a reservoir – GigaOm

  58. Twitter is a stream, but it’s also a reservoir http://t.co/P3RgkBlm

  59. RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  60. Ten FM 89.7 and 98.7 Saturday, November 19, 2011

    RT @2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/5DN3JlJ9 /@mathewi @Gigaom *great read!*

  61. RT @2morrowknight #Twitter is a stream, but it’s also a reservoir http://t.co/XrUyxbsV /@mathewi @Gigaom *great read!*

  62. Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/cDgpJF1A

  63. Maggie Rawlinson Saturday, November 19, 2011

    RT @jeffreypjacobs: Twitter is a stream, but it’s also a reservoir — Tech News and Analysis http://t.co/cDgpJF1A

  64. Kathy Meyer  Saturday, November 19, 2011

    #Twitter is a stream, but it’s also a reservoir http://t.co/qFzHngBQ /@mathewi @Gigaom *great read!* RT @2morrowknight

  65. RT @2cre8: #Twitter is a stream, but it’s also a reservoir http://t.co/qFzHngBQ /@mathewi @Gigaom *great read!* RT @2morrowknight

  66. Chris McLaughlin Saturday, November 19, 2011

    Twitter is a stream, but it’s also a reservoir- http://t.co/1DAXG57R #SocialMedia

  67. #Twitter is a stream, but it’s also a reservoir http://t.co/5YnfvEch /@mathewi @Gigaom @2morrowknight

  68. “@2morrowknight: #Twitter is a stream, but it’s also a reservoir http://t.co/M6u39V8G /@mathewi @Gigaom *great read!*”

  69. Twitter is a stream, but it’s also a reservoir http://t.co/5DNtF903 nvia @mathewi

  70. RT @Digitaltonto: Twitter is a stream, but it’s also a reservoir http://t.co/fCV0KdjG nvia @mathewi

  71. Teresa Di Cairano Sunday, November 20, 2011

    RT @Digitaltonto: Twitter is a stream, but it’s also a reservoir http://t.co/5DNtF903 nvia @mathewi

  72. Zain Abdurrahman Sunday, November 20, 2011

    RT @gigaom: Twitter is a stream, but it’s also a reservoir http://t.co/iY4URdu1

  73. Twitter Activity Stream & Dwindling Attention Reservoir http://t.co/RmE2GM1O Twitter is a Stream & Reservoir http://t.co/wfDmvu8B @mathewi

  74. RT @gigaom: Twitter is a stream, but it’s also a reservoir http://t.co/iY4URdu1

  75. Kameron Hansen Sunday, November 20, 2011

    Twitter is a stream, but it’s also a reservoir – GigaOm http://t.co/nTlaUdCc

  76. Chris McLaughlin Sunday, November 20, 2011

    Twitter is a stream, but it’s also a reservoir– http://t.co/1DAXG57R #SocialMedia

  77. RT @mclaughlinchris: Twitter is a stream, but it’s also a reservoir– http://t.co/1DAXG57R #SocialMedia

  78. Interesting piece about the emerging community of startups that do analysis on big #twitter data http://t.co/T1d8RiP5 #bigdata #ecosystem

  79. RT @dombeveridge: Interesting piece about the emerging community of startups that do analysis on big #twitter data http://t.co/T1d8RiP5 #bigdata #ecosystem

  80. Conor O’Neill Monday, November 21, 2011

    RT @dombeveridge: Interesting piece about the emerging community of startups that do analysis on big #twitter data http://t.co/T1d8RiP5 #bigdata #ecosystem

  81. Tools for capturing and archiving the Twitter stream. Could make Twitter even more valuable. http://t.co/qWcg9ONT

  82. Anne-Laure Thomas Tuesday, November 22, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/6pDMf14B

  83. @Twitter is a stream, but it’s also a reservoir #data #revolution http://t.co/0WBJnB6B

  84. Felicity Thinks Tuesday, November 22, 2011

    Twitter is a stream, but it’s also a reservoir (via @swee06840) – http://t.co/xsdBcn2j

  85. Christophe Lejeune Tuesday, November 22, 2011

    Twitter is a stream, but it’s also a reservoir http://t.co/6XCC6tWA

  86. #Twitter is a stream, but it’s also a reservoir http://t.co/UwKkZNPg rt @mathewi @Gigaom @2morrowknight

  87. Twitter is a stream, but it’s also a reservoir http://t.co/5bpZFXCn

  88. http://t.co/nYCpaWTc Twitter is a stream, but it’s also a reservoir – GigaOm

  89. http://t.co/nTlaUdCc Twitter is a stream, but it’s also a reservoir – GigaOm

  90. Twitter is a stream, but it’s also a reservoir – GigaOm http://t.co/nYCpaWTc

  91. Twitter is a stream, but it’s also a reservoir http://t.co/kYS0NMUI RT @boalt #SocialMedia #patterns

  92. Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  93. Régia Valléria Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  94. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  95. Louise Pereira Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  96. Rodrigo Édipo Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  97. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  98. Colby Stuart • QBBV Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  99. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  100. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  101. ricardo k sulzbach Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  102. teresa benassi Monday, November 28, 2011

    RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  103. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  104. Twitter is a stream, but it’s also a reservoir – GigaOm http://t.co/nYCpaWTc

  105. RT @plevy: Twitter is a stream, but it’s also a reservoir http://t.co/J9EFXTfl via @younaz #UOKM

  106. Twitter is a stream, but it’s also a reservoir – GigaOm http://t.co/Nifkdq9q

  107. http://t.co/nYCpaWTc Twitter is a stream, but it’s also a reservoir – GigaOm

  108. Kimberly Stinson Sunday, December 11, 2011

    RT @mathewi: in case you missed it yesterday: “Twitter is a stream, but it’s also a reservoir” http://t.co/Ax9c0E0L

Comments have been disabled for this post