Blog Post

Is your big data worth the effort?

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

Building big databases of esoteric data may seem like a noble cause, but is it really worth it? That’s the question any company IT manager or data scientist should be asking themselves, Warner Music Group SVP Dave McCrory said at GigaOM’s Structure conference.

Big data usually yields small bits of useful of information, he said. Sometimes those small bits of information can be very valuable, but ultimately there’s an economics calculation that has to be made: is the expense of collecting, storing and crunching that data to extract that information actually smaller than the value it brings your company?

“I’ve come across several people that are collecting data just to collect because at some point in the future they might be able to get some valuable information out of it,” McCrory said. “The question is, is it worth doing that? Is it worth keeping that data and for how long? … You can’t just store your data forever. It’s not free to do, and it’s a compounding problem.”

Bottom line: building massive Hadoop clusters is expensive. You’d better be sure the effort is justified. McCrory likened data mining to actual mining: In mining shale oil the actual cost of extracting petroleum can actually exceed the value of the oil itself.

Check out the rest of our Structure 2013 live coverage here, and a video embed of the session follows below:

[protected-iframe id=”05b8ffc952be801db9f5899f0a4ddd03-14960843-61002135″ info=”″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]
A transcription of the video follows on the next page

2 Responses to “Is your big data worth the effort?”

  1. Matt Fates

    Clearly an important question to ask when spending big dollars on IT projects…”Is it worth it?” But in this case, that leads to a number of other questions, such as “If you could take advantage of all available data related to a specific problem or opportunity, would you be able to make better decisions?” and “Is gathering all the data easy? hard? reasonable?”; “Do you already have a lot of the data, but it is silo-ed or dispersed?” I certainly agree that Big Data is all the rage at the moment, and some companies are likely deploying it in a way that is total overkill…but they are also learning from their projects and attempts…and this will be valuable. And the costs will come down. As it becomes cheaper and easier to leverage data, those who have started early will have an advantage as they have been investing resources in figuring it out ahead of the crowd.

  2. Ah, but Big Data and its mining is the vogue thing to do, right? Oh and by the way, why is it that many a (particularly Silicon Valley) company finds the need to brag about their use of big data and its mining? Are these companies really that insecure about themselves they find the need to show off (and in the process justify themselves to their VC investors)?