Comments on: The big data world is operating at 1 percent

By: Jack Rivkin

Jack Rivkin — Wed, 13 Mar 2013 00:09:42 +0000

Can’t wait until we are using more than 1% of the data available and accumulating. There is a “law” that applies here: “The number of conclusions reached expands proportionately to the data available and inversely to the time required to reach a conclusion.” What one measures or analyzes affects behavior. It does start with strategy and what one is trying to accomplish. Big Data offers some exciting opportunities, but our history of data analysis says “proceed with caution.” See “What is the Big Deal About Big Data?” for some more thoughts on this: http://bit.ly/YCd7DE

By: gurjeets

gurjeets — Mon, 11 Mar 2013 22:33:23 +0000

Thanks to everyone for posting such thoughtful comments. I find it interesting that several of you pointed out the value of queries. I agree that queries are important, but only after the entire data set is algorithmically mapped into a topology that a data scientist, or business user, can begin with. In other words, once the relationships are mapped across the entire dataset, then the user can “zoom into” specific areas of the visualization to explore the meaning of shape and color. It’s about starting with a machine-generated view of the data that is unbiased and holistic (using 100% of the data). Thomas is correct when he says that “starting with queries is a dead ethos that gets increasingly meaningful as we approach huge datasets and very complex correlated sciences that no on has yet dreamt about a possible hypothesis to start with.” We are in the first inning of the Big Data game and I look forward to what we will all discover when we explore new approaches.

By: arjkay

arjkay — Mon, 11 Mar 2013 14:21:12 +0000

Here here! Insights are harder to derive when the ingrained assumptions of causality lead us to look for affirming evidence, rather than understanding more fully the interactions, or impacts of different event chains.
The problems Gurjeet raises reflect the persistence of linear thinking and heuristic bias. The opportunities that Genetic Algorithms, or self-learning models present parallels processing that occurs in the human brain requires additional automation to make the dynamic feedback or output manageable. I’m all in favor of adding new tools that allow us to reliably reconstruct scenarios and iteratively discover what doesn’t fit the expected patterns. What makes us well, may be different than what makes us sick. The timing of data capture, which accounts for why we may be only at 1%, doesn’t have to be a limitation. To the contrary it’s proven useful to sample.
The difference between data capture and analysis to date and what the future holds hinges on correctly applying extrapolation methods. Only when life and death are at stake, does eliminating the confidence interval in your sample based prediction warrant inclusion of more data.
In a dynamic world, certainty may be over-rated and leaving some room for doubts will help us continue to keep looking, probing and discovering.

By: Thomas Chacko

Thomas Chacko — Mon, 11 Mar 2013 08:59:27 +0000

Agree to this mostly Gurjeet..We have enough historical data available in some form within specific domain datastores. This can be mined by tools from Ayasdi and others to usher in critical breakthroughs in cancer research, energy exploration, drug discovery, financial fraud detection and more. However most entities, unless mandated by regulation OR not involved in any traditional data mining and/or piloting a big data project – attach a very high cost with data retention. With the result, most of these valuable data keeps spilling outside the retension-window and lost for ever for any future application.

“Starting with queries is a dead end” — this ethos gets increasing meaningful as we approach huge datasets and very complex correlated sciences where no one as yet dreamt about a possible hypothesis to start with. The caveat being that we are to identify and steer clear of ‘false positive’ traps and be able to verify the outcome confidently using multiple approaches.
Nevertheless, a very interesting era we are all staring at..

By: Nico

Nico — Mon, 11 Mar 2013 05:44:55 +0000

The most important aspect to big data is the company strategy and how the company is willing to align to the insights from big data. First they must have a strategy and know what they want from the data, and “more sales” is not valid. Then, when they have the results they must be able to act on those results, make decisions and be more flexible in implementing those decisions.

By: Schills

Schills — Mon, 11 Mar 2013 02:58:11 +0000

I am not a Sara scientist by any stretch of the imgaination so forgive me if I sbuse the terminology. Queries are not valuable solely for finding positives. Their real value lies in finding negatives, and eliminating those from the data being considered.

By: cappetit60

cappetit60 — Sun, 10 Mar 2013 23:02:55 +0000

Entrada de pruebas hoy primer día .

By: Michael Brill

Michael Brill — Sun, 10 Mar 2013 20:37:17 +0000

While I understand the sentiment behind “starting with queries is a dead end,” I think that’s exactly where you need to start. More precisely, getting to user intent is critical. Otherwise, we’ll continue down this path where our fancy analytics just generate a steady stream of false-positives and otherwise non-actionable output.

Maybe the trick is creating software that can elicit queries from humans in a structure that is more usable than simple database queries against fixed schema. This gives machines and data scientists something market-driven to aim for rather than guessing what the market might way… kind of lean principles applied to analytics.

By: Mark McAndrew

Mark McAndrew — Sun, 10 Mar 2013 17:47:52 +0000

Hear, hear.

It’s not just 99% of data that isn’t being used. Same applies to the world’s total computing power. Put them together, it really will change the world: http://www.charityengine.com/news/blogs/charity-engine-mission