That needle in the haystack of useful big data may be smaller than we thought

needle in haystack

New research analyzing the log data churned out by applications developed on the Heroku platform as a service shows just how little of that data is actually useful to developers or devops personnel running those applications.

logentries2Out of 22 billion log events across 6,000 Heroku applications, just 0.18 percent held information that a developer (or a devops pro) would actually need to know to prevent a failure or boost performance, according to Logentries. Taken another way, 99.82 percent of that data is the haystack of non-useful stuff.

Of course, Logentries is publicizing this because it’s in the business of helping people winnow out that 0.18 percent without having to sift through the rest, but still, it’s interesting.

Boston-based Logentries last month netted $10 million in venture funding to take on Splunk and Sumo Logic and make big data understandable to civilians (i.e., non-data scientists.)

Logentries claims that its SaaS service “pre-processes” data as it’s generated so that sussing out insights can happen faster.

loading

Comments have been disabled for this post