1 Comment

Summary:

The Sunlight Foundation, a non-profit aimed at showing how corporate interests influence government released a pretty sweet tool for citizens and big data nerds on Monday. Looking at Capitol Words, it’s easy to see how big data and cheap computing could improve government accountability.

sunlight foundation thumb

The Sunlight Foundation, a non-profit aimed at showing how corporate interests influence government released a pretty sweet tool for citizens and big data nerds on Monday. The tool, called Capitol Words, monitors how often, and which, legislators said certain phrases in an effort to track how those phrases enter and influence the political debate. Capitol Words is one of those random tools that gives us a glimpse of how cheap computing and better data analytics can change the business as usual in politics.

Already, the combo of cheap computing and big data are changing how retail firms set prices, offering insights into healthcare, and helping investors maximize rental income, so the notion it could lead to more government transparency isn’t all that crazy. In the case of Capitol Words and the Sunlight Foundation, the goal is to analyze what legislators say on the floor of the House and Senate to track how an idea can filter through a political party, a region or a debate by parsing the text data generated daily by the Congressional Record.

Tom Lee, the Director of Sunlight Labs at the Sunlight Foundation, said in an interview that the amount of data isn’t huge — about 50 or 60 gigabytes a day — but the text does need to be parsed so it can be made into something useful. So the Sunlight Foundation has developed algorithms and techniques, many of which it releases on Github, for using the data. It does the calculating and analysis on Amazon’s Elastic Map Reduce service and then uses Solr, an open-source search platform, to process people’s queries against the records. The database supporting the tool has upwards of 20 million records.

“The speech used by legislators is used to advance causes and manipulate the public,” Lee said. “And how their speech is similar or different can show how particular terms originate from some political messaging memo.”

Lee said the original version of the project in 2008 ran the search and the data parsing in parallel, but that approach was too compute-intensive and didn’t allow for the richness of the results the project can offer today by splitting the two steps up. However, he didn’t rule out coming back to running the job in parallel eventually as the data stores become larger and queries became more complex.

The Sunlight Foundation makes its findings and data available via a JSON API so others can build on it. It’s also hoping to expand beyond floor speeches to politician’s appearances on talk shows and other venues. It hopes to create other services by tying these political sound bytes to its repository of funding data, which tracks what lobbying groups and individuals politicians accept money from. It has over a terabyte-and-a-half of data on hand to work from.

And for those eagerly watching how our government attempts to become more transparent and share data, the Foundation is also working with the Government Printing Office, which published the Congressional Record to get the document in a more web-friendly, structured format. That would help the Capitol Words project become more useful and help others build their own data analytics based on what’s said in Congress. Right now, much of the esoteric (somewhat stilted) debate most often hits the general public when The Daily Show  mocks it. While Jon Stewart may be funny, he doesn’t offer the ability to track ideas over time or in any broad fashion.

“For us, this is trying to expand the way Sunlight tracks influence,” Lee said. “We track the way the money flows around Washington and it’s not enough. The ways the system is affected are too subtle and deliberate, so we’re making an investment in tracking not just the flow of money but also the flow of ideas.” And when you’re trying to track something as nebulous as ideas, analyzing a lot of data using cheap compute is perhaps the only way for a non-profit to do it.

You’re subscribed! If you like, you can update your settings

  1. [DocDocc] Non-profit uses big data to track big government http://t.co/0UnAYSoJ via gigaOM

  2. Non-profit uses big data to track big government http://t.co/X4AQNumW

  3. Non-profit uses big data to track big government http://t.co/cYYwFYvN #Cloudcomputing

  4. Jacquelyn Jenkins Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/NibfBtrU #cloud #gigaom

  5. amazinginteriors Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/aguLWI4w

  6. Non-profit uses big data to track big government http://t.co/tRix1hPZ

  7. Non-profit uses big data to track big government http://t.co/ewnenu9b

  8. Non-profit uses big data to track big government http://t.co/FLez61JO @GigaOM

  9. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how… http://t.co/GrKU9pUW

  10. Yasbella Sebastian Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/xTp0XutW

  11. Non-profit uses big data to track big government http://t.co/KauI23DW

  12. Non-profit uses big data to track big government http://t.co/tjbGf9Mv

  13. Non-profit uses big data to track big government http://t.co/E6QgDPCQ

  14. Non-profit uses big data to track big government http://t.co/uAlN2caR

  15. Non-profit uses big data to track big government http://t.co/1NbpCuc5

  16. Derrick Harris Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/pF43zxX8 by @gigastacey < “tracking not just … money but also … ideas.”

  17. MAKE GOOD MONEY Monday, December 12, 2011

    PROFIT NEWS! Non-profit uses big data to track big government http://t.co/01mEBFHJ

  18. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed… http://t.co/KQXWWM4p

  19. Luca Filigheddu Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/T5zzBGpJ

  20. Nicholas Frota Monday, December 12, 2011

    Non-profit uses big data to track big government: GigaOM on Sunlight Foundation. http://t.co/TY36LN9a

  21. Non-profit uses big data to track big government http://t.co/LygtiNPP via Stacey Higginbotham

  22. Non-profit uses big data to track big government http://t.co/xHsGd8qu

  23. .@SunFoundation uses #bigdata to track big government http://t.co/Xceo1d4F

  24. Non-profit uses big data to track big government http://t.co/IF2qqNgw #marketing #business #success #biztips

  25. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  26. gigaom: Non-profit uses big data to track big government http://t.co/x95FIcvK

  27. Amazing to see so many Big Data applications emerge “Non-profit uses big data to track big government” http://t.co/CjViNBdN

  28. Non-profit uses big data to track big government http://t.co/NO4pZU3b

  29. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/3GfywOEZ

  30. Non-profit uses big data to track big government http://t.co/d4grchkt

  31. Non-profit uses big data to track big government http://t.co/CAKAASGP @amarchugg #news

  32. Crunked On Tech Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/xIrdVGbb

  33. Non-profit uses big data to track big government http://t.co/wsZdghTS @amarchugg #news

  34. Ray Informatics Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/vcoN3FkB

  35. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/ONjpYotS

  36. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/Ui3yAgqN

  37. Roberts Chapman Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/RVf4ME0G

  38. Peter Alexandrov Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/QHerel2s

  39. Non-profit uses big data to track big government: http://t.co/ER9AYKUQ via @gigaom (attn: @antheawatson)

  40. re: @SunFoundation cc @EllnMllr @sunlightonhill // RT @gigaom: Non-profit uses big data to track big government http://t.co/NN1ppFeo

  41. Non-profit uses big data to track big government: 12, 2011, 4:00pm PT No Comments The Sunlight Foundation, a non… http://t.co/bFqwwgJj

  42. Anthea Watson Strong Monday, December 12, 2011

    Great piece about @sunlightfoundation and their use of big data to track government http://t.co/9Dxm9caA via @gigaom & @conniehwong

  43. RT @antheawatson: Great piece about @sunlightfoundation and their use of big data to track government http://t.co/9Dxm9caA via @gigaom & @conniehwong

  44. RT @antheawatson: Great piece about @sunlightfoundation and their use of big data to track government http://t.co/9Dxm9caA via @gigaom & @conniehwong

  45. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  46. Venkat Viswanathan Monday, December 12, 2011

    RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  47. Non-profit uses big data to track big government: 12, 2011, 4:00pm PT No Comments The Sunlight Foundation, a non… http://t.co/FE06styT

  48. Non-profit uses big data (EMR) to track big government http://t.co/p7MJulhT #aws #elasticmapreduce

  49. CloudConnexion Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed… http://t.co/ZTpZdk4h

  50. Liz Bartolomeo Monday, December 12, 2011

    Non-profit (ie. @SunFoundation) uses big data to track big government http://t.co/TWMr5ocs

  51. Daniele Russolillo Monday, December 12, 2011

    RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  52. Richard Campione Monday, December 12, 2011

    Non-profit #Sunlight uses #BigData to track how corporate interests influence big #government http://t.co/eAdJjyoW v @GigaOM

  53. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  54. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  55. Non-profit uses big data to track big government http://t.co/uNGus83n

  56. Denisse _Matsuno Monday, December 12, 2011

    Non-profit uses big data to track big government: Capitol Words is one of those random tools that gives us a gli… http://t.co/UgsjDOle

  57. Nathaniel Flick Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/XJh3rSpZ

  58. A tool for tracking the spread of the latest disinformation from lobbyists via Congress: http://t.co/vXTGjIB1

  59. Non-profit uses big data to track big government http://t.co/VX8UzxNn

  60. RT @gigaom: Non-profit uses big data to track big government http://t.co/SUj0e5Bp

  61. Non-profit uses big data to track big government: 12, 2011, 4:00pm PT No Comments The Sunlight Foundation, a non… http://t.co/qZpTekko

  62. #cloud Non-profit uses big data to track big government – The Sunlight Foundation, a non-profit aimed at showing how… http://t.co/QOG48QR2

  63. Non-profit uses big data to track big government http://t.co/vd7tofke via @gigaom > one reason I love #BigData, Big Brother backwards. #in

  64. Tenay Hankins Non-profit uses big data to track big government: It hopes to create other se… http://t.co/Dz23jOu6 http://t.co/esEmgvdO

  65. RT @gigaom: Non-profit uses big data to track big government http://t.co/i9BTilkZ

  66. Non-profit uses big data to track big government http://t.co/ehOVsoL1

  67. Charles Chuang Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/jPPste4H via @zite

  68. Non-profit uses big data to track big government http://t.co/KzGoyDgS via @zite

  69. Network Reading Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/bPR0Nfj9

  70. Stephen Clifford Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/PYNQ7lTm

  71. Mahesh P-Subramanya Monday, December 12, 2011

    Non-profit uses big data to track big government http://t.co/qfFJ2RPr

  72. Non-profit uses big data to track big government http://t.co/7EXni4Xc

  73. Olafur Ingthorsson Monday, December 12, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aim… http://t.co/yP2OnbTq CloudComputingTopics

  74. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  75. Good story on @SunFoundation Capitol Words project: http://t.co/eN5UcOKA

  76. Fed. Priorities Data Monday, December 12, 2011

    Just caught the @gigaom writeup of @sunlight labs Capitol Words. Congrats! http://t.co/EopDKCFE

  77. Big Data Technology Monday, December 12, 2011

    Non-profit uses big data to track big government – http://t.co/sm9qtTeW (db)

  78. Capitol Words from @sunlightlabs: monitoring the hot air of legislators. Now that there is some big data! http://t.co/b5qnoG7u

  79. Hell Yes! In addition to this, I’d love to see bills interpreted into layman’s terms for the average American.

  80. Yasunori Matsuki Monday, December 12, 2011

    RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  81. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  82. Non-profit uses big data to track big government http://t.co/JppiUjZs shared via NewsMix http://t.co/EURKeXt3 (hat tip @gigaom)

  83. Stacey Higginbotham Monday, December 12, 2011

    RT @derrickharris: Non-profit uses big data to track big government http://t.co/pF43zxX8 by @gigastacey < “tracking not just … money but also … ideas.”

  84. Non-profit uses big data to track big government http://t.co/Bx79mFIJ

  85. Nathania Johnson Monday, December 12, 2011

    Non-profit uses big data to track government http://t.co/KWFWgm8m

  86. Non-profit uses big data to track big government – GigaOm http://t.co/mxKZVqfm

  87. Non-profit uses #bigdata to track big government — http://t.co/8Fbkg9mY

  88. LatentView Analytics Monday, December 12, 2011

    RT @HadoopNews: Non-profit uses #bigdata to track big government — http://t.co/8Fbkg9mY

  89. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  90. RT @gigaom: Non-profit uses big data to track big government http://t.co/jZ9HMsl4

  91. RT @techpower: Amazing to see so many Big Data applications emerge “Non-profit uses big data to track big government” http://t.co/CjViNBdN

  92. NGO uses #bigdata to track #gov. @SunFoundation’s tool monitors how often & which legislators say certain phrases http://t.co/Z015bsUu

  93. Non-profit uses big data to track big government http://t.co/oGuJR58I

  94. Non-profit uses big data to track big government — Cloud …: The Sunlight Foundation, a non-pro… http://t.co/69a0xB9F #bigdata #blogs

  95. love big data “@techpower: …so many Big Data applications emerge “Non-profit uses big data to track big government” http://t.co/mDamszw9”

  96. Non-profit uses big data to track big government http://t.co/w4LhJuEw

  97. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  98. RT @gigaom: Non-profit uses big data to track big government http://t.co/tRix1hPZ

  99. Non-profit uses big data to track big government http://t.co/xKULeCJg via @zite < tracking ideas in US politics as well as money

  100. Caitlin Fitzsimmons Tuesday, December 13, 2011

    RT @kevglobal: Non-profit uses big data to track big government http://t.co/xKULeCJg via @zite < tracking ideas in US politics as well as money

  101. Non-profit uses big data to track big government – http://t.co/4eMIIfNQ “For us this is trying to expand the way Sunlight tracks influence”

  102. Cool new tool from Sunlight Foundation to track the government with data: http://t.co/Abuifg3G #itpol /via @glynmoody /cc @benteka

  103. Non-profit uses big data to track big government — Cloud …: The Sunlight Foundation, a non-profit aimed at sho… http://t.co/SX13Q7S6

  104. Non-profit uses big data to track big government http://t.co/2qA6ZXiN

  105. http://t.co/dNQt2aub Non-profit uses big data to track big government – GigaOm

  106. Ruud Steltenpool Tuesday, December 13, 2011

    RT @glynmoody: Non-profit uses big data to track big government – http://t.co/4eMIIfNQ “For us this is trying to expand the way Sunlight tracks influence”

  107. Non-profit uses #bigdata to track big government http://t.co/xL6sdM3D via @giagaom #data

  108. GigaOM: Non-profit uses big data to track big government http://t.co/eF3AHhd3

  109. RT @techpower: Amazing to see so many Big Data applications emerge “Non-profit uses big data to track big government” http://t.co/CjViNBdN

  110. RT @Techpower Amazing – so many #BigData apps emerging < “Non-profit tracks big govt using big data” http://t.co/BrzS83yJ [via @BigDataExpo]

  111. RT @digimindci Non-profit uses big data to track big government http://t.co/byvOpDve

  112. Políticos controlados por la palabra RT @gigaom: Non-profit uses big data to track big government http://t.co/z0yU3vSi

  113. MAKE GOOD MONEY Tuesday, December 13, 2011

    PROFIT NEWS! Non-profit uses big data to track big government — Cloud … http://t.co/0JBdR8La

  114. Non-profit uses big data to track big government http://t.co/UBmSR5We

  115. Non-profit uses big data to track big government http://t.co/fJWItRbr via @zite #opendata #opengov #gov20

  116. RT @jkonga: Non-profit uses big data to track big government http://t.co/fJWItRbr via @zite #opendata #opengov #gov20

  117. Non-profit uses big data to track big government http://t.co/as7I6gNl

  118. George T. Thibault Tuesday, December 13, 2011

    Non-profit uses big data to track big government http://t.co/csVturbb

  119. Non-profit uses big data to track big government http://t.co/cI98Iuv4

  120. Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/4ZaE5ukW

  121. Roberts Chapman Tuesday, December 13, 2011

    Non-profit uses big data to track big government: The Sunlight Foundation, a non-profit aimed at showing how cor… http://t.co/hXsgQ9tq

  122. [317] Non-profit uses big data to track big government
    http://t.co/IUpaC228

  123. Stacey Higginbotham Tuesday, December 13, 2011

    Non-profit uses big data to track big government http://t.co/X1n2TDnI >> New ideas on holding politicians accountable using data.

  124. RT @gigastacey: Non-profit uses big data to track big government http://t.co/X1n2TDnI >> New ideas on holding politicians accountable using data.

  125. Non-profit uses big data to track big government http://t.co/HZOc2tYN

  126. RT @gigastacey: Non-profit uses big data to track big government http://t.co/X3KSgXA4 >> I’m curious to see the “win-win” meter

  127. “Non-profit uses big data to track big government”-@GigaOM http://t.co/tOuiOGI3 @gigastacey on “Capitol Words.” #opengov #gov20

  128. Non-profit uses big data to track big government: http://t.co/Cfbs6aa0 HT @digiphile

  129. RT @gigastacey: Non-profit uses big data to track big government http://t.co/X1n2TDnI >> New ideas on holding politicians accountable using data.

  130. RT @digiphile: “Non-profit uses big data to track big government”-@GigaOM http://t.co/tOuiOGI3 @gigastacey on “Capitol Words.” #opengov #gov20

  131. “@digiphile: “Non-profit uses big data to track big government”-@GigaOM http://t.co/ZW9Q6UBx “#opengov #gov20” @ComputerNewsME

  132. RT @PallaviSharma5: “@digiphile: “Non-profit uses big data to track big government”-@GigaOM http://t.co/ZW9Q6UBx “#opengov #gov20” @ComputerNewsME

  133. RT @gigastacey: Non-profit uses big data to track big government http://t.co/X1n2TDnI >> New ideas on holding politicians accountable using data.

  134. Non-profit uses big data to track big government http://t.co/JAzVIvYG

  135. On my reading list: http://t.co/Mw0jo63V Non-profit uses big data to track big government

  136. Mythic Technologies Tuesday, December 13, 2011

    Non-profit uses big data to track big government – http://t.co/o4UZ8Yay

  137. Shelly Alcorn, CAE Tuesday, December 13, 2011

    Non-profit uses big data to track big government http://t.co/0TpkNg9h <wow!>

  138. RT @iDeanUToronto: Non-profit uses big data to track big government http://t.co/Bx79mFIJ

  139. Abdul Jaleel KK Tuesday, December 13, 2011

    Non-profit uses big data to track big government (GigaOM) http://t.co/7K6tiuVs

  140. Abdul Jaleel KK Tuesday, December 13, 2011

    Non-profit uses big data to track big government (GigaOM) http://t.co/7K6tiuVs

  141. Non-profit analyzes political influence of House and Senate by parsing text of Congressional Record http://t.co/Wtu1jH0c #bigdata

  142. Interesting piece on @SunFoundation: “Non-profit uses big data to track big government” http://t.co/QPxRXHIU

  143. RT @CorbinHiar: Interesting piece on @SunFoundation: “Non-profit uses big data to track big government” http://t.co/Xbo5vB8Y

  144. @dandrinkard 60g new data per day, per http://t.co/rCQwuym9 ? Seemed big. Was wondering what the full data sets were @antheawatson @nekaro

  145. Non-profit uses big data to track big government — Cloud … http://t.co/Kw6ArOt9

  146. Bigdata use case :Non-profit uses big data to track big government http://t.co/CiPaBPlJ #bigdata #EMR #solr

  147. Trying to have more transparent governments. Non-profit uses big data to track big government http://t.co/JarSOOBq

  148. Cool! @SunFoundation uses data mining 2 compare legislator speech 2 corporate phrases 2 measure influence. http://t.co/pjrp51m3

  149. Great article about @SunFoundation http://t.co/lNStPgcn via @iDeanUToronto

  150. Non-profit uses #big #data to track big government — Cloud Computing News | http://t.co/oOVJqhm3

  151. Sybase Analytics Tuesday, December 13, 2011

    @gigastacey looks at #BigData & big #government in @GigaOm check out how the non profits are deploying #analytic tools http://t.co/ew1yCLpd

  152. 非営利団体、ビッグデータを用いてビッグな政府を監視
    http://t.co/yjFoxaNi

    企業の関心がどのように政府に影響を与えているかを示すことを目的 とする非営利団体Sunlight Foundationは12日、市民に向けた非常に 簡単に扱えるツールを発表した。

  153. Anthony Russell Tuesday, December 13, 2011

    Non-profit uses big data to track big government — Cloud Computing News http://t.co/l0MpaLYG

  154. RT @MarkDalgarno: Non-profit uses big data to track big government Cloud Computing News http://t.co/X0BgqNei #Cloud

  155. Pan de Vida Fnd – QC Tuesday, December 13, 2011

    Non-profit uses big data to track big government http://t.co/OZazQBwh

  156. Non-profit uses big data to track big government ( Stacey Higginbotham ) ::: http://t.co/np8HC3sX :: #BigData

  157. Rivka Gewirtz Little Wednesday, December 14, 2011

    RT @gigabarb: Non-profit uses big data to track big government http://t.co/as7I6gNl

  158. Non-profit uses big data to track big government http://t.co/1Ypm9hbQ

  159. Centered Networks Wednesday, December 14, 2011

    Non-profit uses big data to track “idea flow” in big government http://t.co/SgQeFFK2 #nonprofit #data #transparency #awesome

  160. Non-profit uses big data to track “idea flow” within big government http://t.co/vVRvfSlg #cloud #awesome

  161. Non-profit uses big data to track big government http://t.co/8NIm4TZt

  162. Mashie Macharia Thursday, December 15, 2011

    Non-profit uses big data to track big government http://t.co/cQdBOFvh

  163. We need this in the UK! RT @joannamikhail Non-profit uses big data to track big government http://t.co/POYY8pNy

Comments have been disabled for this post