Take Recorded Future, which mines data from the web, social media, and government and financial sites to create reports around topics like early signs of food and water shortages. The reports analyze how those conditions lead to unrest, terrorism and conflict. For example, Recorded Future wrote a blog post a couple years ago showing the link between crop shortages, lack of food, extreme weather and political instability in Yemen.

As Recorded Future’s CEO Chris Ahlberg explained to me in an interview, “Hungry people will do all kinds of things.” A substantial part of Recorded Future’s customers are governments and folks in the intelligence community, so Ahlberg couldn’t go into many specifics, but he said that he could envision organizations like World Banks, also using similar big data info to create aid and policies.

Other companies are using Twitter data, in particular — leveraging its real-time nature — to track the pain points in population growth, such as the spread of diseases. A recent study by medical researchers at Harvard showed that Twitter was substantially faster at tracking the spread of cholera in Haiti than more traditional methods (this was in the wake of the earthquake, but you can imagine the same techniques could be used on a regular basis).

Cell phone data can be particularly valuable for studying population growth. Because cell phones tend to be with people all the time and billions of people in the world have cheap cell phones, they can often stand in for location data. Ahlberg says Recorded Future can, for example, isolate its analysis to pulling data from only mobile devices in Yemen.

A project developed by Nathan Eagle, a professor at Harvard’s School of Public Health and MIT Media Lab, is compiling millions of phone records to help development groups make important decisions like finding the best region in Kenya to launch a malaria eradication campaign or enabling health care providers to discover abnormal patterns in cholera outbreaks in Rwanda.

Finally, big data analytics can also help with management of resources directly, such as smarter water systems. IBM has been a leader in this space, and Opower, a venture-backed energy software startup, told me that it has been transitioning to using Hadoop, via startup Cloudera, to run heavy analytics on the utility energy data it crunches in the cloud. Sensors and analytics can lead to much smarter cities, buildings and transportation (for more info on that read our report for GigaOM Pro, “Key Technologies for the Future of the Smart City”, subscription required).

Image courtesy of Vinoth Chandar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Key technologies for the smart city
• A near-term outlook for big data
• Ups and downs for cleantech in Q1

(c) 2012 Pinar Ozger. pinar@pinarozger.com

This ability to monetize unproductive assets, is a huge selling point for big data, said Paul Speciale, VP of products at Amplidata, the object storage company that worked with the festival on that project. So is the ability to look outside your company to see and analyze what users, would-be users, and competitors are saying about your products and services — thus all the talk about analyzing the Twitter firehose and Facebook data.

But such projects are fueling expectations for more, better, and faster big data interactions, according to speakers at the GigaOM Structure:Data 2012 event in New York on Wednesday. The advent of consumer technologies like the iPhone’s Siri have educated consumers about the need and application of natural language processing. The ability to handle  unstructured speech is a key component of many big data applications.

“We are past the distinction between consumer and business,” said Staffan Truve, CTO and co-founder of Recorded Future. ”They drive each other.”

Jason Hunter, deputy CTO of MarkLogic, agreed that the explosion of fast, powerful consumer devices is driving demand for better big data applications. “I remember waiting for [compute] jobs to process over night. Now if I’m not sure I’m getting 60-frames-per-second on my iPad, I’m upset. Expectations change. I want lots of data, smart data. I want it free and I want it pretty.”

This exploding demand means the technologies around outputting data in a useable, understandable format, ingesting it into storage so that it’s manageable and searchable, and the analytics to  parse that data so it’s useable will only grow.

Watch the livestream of the event here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• The importance of putting the U and I in visualization
• A near-term outlook for big data