Comments on: Why Big Data & Real-Time Web Are Made For Each Other http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/ Trusted Insights and Conversations on the Next Wave of Technology Mon, 23 Nov 2009 04:23:31 +0000 http://wordpress.com/ hourly 1 By: Big Data and Real-time Structured Data Analytics http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-967034 Big Data and Real-time Structured Data Analytics Thu, 13 Aug 2009 21:26:13 +0000 http://gigaom.com/?p=55641#comment-967034 [...] data in near real-time. But as Truviso founder and UC Berkeley CS Professor Michael Franklin recently noted, there are mountains of structured data generated by web apps that lend themselves to real-time [...] [...] data in near real-time. But as Truviso founder and UC Berkeley CS Professor Michael Franklin recently noted, there are mountains of structured data generated by web apps that lend themselves to real-time [...]

]]> By: 9thpoint http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-959264 9thpoint Mon, 13 Jul 2009 16:17:09 +0000 http://gigaom.com/?p=55641#comment-959264 The real problem with real time data analytics is that people cannot react to it fast enough. At the end of the day these streams of data will make no sense if companies do not have the time to react to it. That process involves at times a number of departments and coordinating an action through these take time. It may force different organization structures but I am no org theorist so I will leave it at that but as things stand right now reacting to "real time" data is like timing the stock market v/s making long term bets based on sound analysis. The former may sound exciting but the latter is more sustainable and produces better returns. Do you want to be a stock broker or Warren Buffet... The real problem with real time data analytics is that people cannot react to it fast enough. At the end of the day these streams of data will make no sense if companies do not have the time to react to it. That process involves at times a number of departments and coordinating an action through these take time. It may force different organization structures but I am no org theorist so I will leave it at that but as things stand right now reacting to “real time” data is like timing the stock market v/s making long term bets based on sound analysis. The former may sound exciting but the latter is more sustainable and produces better returns. Do you want to be a stock broker or Warren Buffet…

]]> By: Mike http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-959230 Mike Mon, 13 Jul 2009 12:51:49 +0000 http://gigaom.com/?p=55641#comment-959230 The real time data collected must be associated with the application of it. For example, the real-time traffic data collected by the Dash Express had real value to other users (as well as other consumers of traffic data). Twitter searching may mine trends and public opinion reactions to events, but it doesn't serve well as a replacement for Google because there is too much data with too little context. Real-time information is important, but there is still value in organizing it and putting it in context. In military terms, first reports are often wrong - the "fog of war" effect. The real time data collected must be associated with the application of it. For example, the real-time traffic data collected by the Dash Express had real value to other users (as well as other consumers of traffic data).

Twitter searching may mine trends and public opinion reactions to events, but it doesn’t serve well as a replacement for Google because there is too much data with too little context. Real-time information is important, but there is still value in organizing it and putting it in context. In military terms, first reports are often wrong – the “fog of war” effect.

]]> By: Craig Kerstiens http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-959160 Craig Kerstiens Sun, 12 Jul 2009 23:08:19 +0000 http://gigaom.com/?p=55641#comment-959160 Esssentially yes, treating the continuous or real time data as a continuous flow, operate on it as it comes in, then if you need to store it and do deeper analysis you can, but it's separate from the realtime insight process. Esssentially yes, treating the continuous or real time data as a continuous flow, operate on it as it comes in, then if you need to store it and do deeper analysis you can, but it’s separate from the realtime insight process.

]]> By: Ian Kallen http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-959148 Ian Kallen Sun, 12 Jul 2009 21:46:06 +0000 http://gigaom.com/?p=55641#comment-959148 Great post, Michael. Event oriented applications are indeed the new black but the irony isn't lost that in order to do interesting things with the stream, you often have to collect an ocean of data. As stream and big data processing technologies proliferate and are increasingly commoditized, it's clearer that each by themselves offer distinct elements of value but the combination provides the unique and more valuable capacity to find real time signal amidst oceans of noise. Overcoming the concurrency challenges and building great applications with the signals are where all of the fun is. Great post, Michael. Event oriented applications are indeed the new black but the irony isn’t lost that in order to do interesting things with the stream, you often have to collect an ocean of data. As stream and big data processing technologies proliferate and are increasingly commoditized, it’s clearer that each by themselves offer distinct elements of value but the combination provides the unique and more valuable capacity to find real time signal amidst oceans of noise. Overcoming the concurrency challenges and building great applications with the signals are where all of the fun is.

]]> By: TweetStream Promoter http://gigaom.com/2009/07/12/big-data-and-real-time-web-a-confluence-of-streams/#comment-959108 TweetStream Promoter Sun, 12 Jul 2009 16:39:54 +0000 http://gigaom.com/?p=55641#comment-959108 So are you suggesting, having "living" modules that analyzes one particular core stream, and consistently spins off millions of other live streams into space, not necessarily being stored unless someone comes along and wants to start to "mine" this virtual stream for data? I am not even sure what im taking about, lol. So are you suggesting, having “living” modules that analyzes one particular core stream, and consistently spins off millions of other live streams into space, not necessarily being stored unless someone comes along and wants to start to “mine” this virtual stream for data?

I am not even sure what im taking about, lol.

]]>