Building big databases of esoteric data may seem like a noble cause, but is it really worth it? That’s the question any company IT manager or data scientist should be asking themselves, Warner Music Group SVP Dave McCrory said at GigaOM’s Structure conference.
Big data usually yields small bits of useful of information, he said. Sometimes those small bits of information can be very valuable, but ultimately there’s an economics calculation that has to be made: is the expense of collecting, storing and crunching that data to extract that information actually smaller than the value it brings your company?
“I’ve come across several people that are collecting data just to collect because at some point in the future they might be able to get some valuable information out of it,” McCrory said. “The question is, is it worth doing that? Is it worth keeping that data and for how long? … You can’t just store your data forever. It’s not free to do, and it’s a compounding problem.”
Bottom line: building massive Hadoop clusters is expensive. You’d better be sure the effort is justified. McCrory likened data mining to actual mining: In mining shale oil the actual cost of extracting petroleum can actually exceed the value of the oil itself.
Check out the rest of our Structure 2013 live coverage here, and a video embed of the session follows below:
A transcription of the video follows on the next page