- Jeff Hammerbacher, co-founder and chief scientist, Cloudera

Well, something has to pay the bills. Thankfully, there’s also a sweeping trend in the data science world right now around bringing those skills to bear on some really meaningful problems, from the effects of tree pruning to mapping humanitarian crises around the world. I don’t know about you, but I’m willing to sacrifice a little digital privacy if it means saving some lives.

We’ve already covered some of these efforts, including the SumAll Foundation’s work on modern-day slavery and future work on child pornography. Closely related is the effort — led by Google.org’s deep pockets — to create an international hotline network for reporting human trafficking and collecting data. Microsoft, in particular Microsoft Research’s danah boyd, has been active in helping fight child exploitation using technology.

This week, I came across two new efforts on different ends of the spectrum. One is ActivityInfo, which describes itself on its website as “an online humanitarian project monitoring tool” — developed by Unicef and a consulting firm called BeDataDriven — that “helps humanitarian organizations to collect, manage, map and analyze indicators.” That partnership actually seems fairly well established (the ActivityInfo website claims it’s used by more than 75 organizations across more than 15,000 sites), although I came across it via a blog post about why BeDataDriven decided to build the database on Google’s cloud.

ActivityInfo’s map editor.

The other effort I came across is DataKind, specifically its work helping the New York City Department of Parks and Recreations, or NYC Parks, quantify the benefits of a strategic tree-pruning program. Founded by renowned data scientists Drew Conway and Jake Porway (who’s also the host of the National Geographic channel’s The Numbers Game), DataKind exists for the sole purpose of helping non-profit organizations and small government agencies solve their most-pressing data problems. It accomplishes this goal by hosting weekend-long DataDives — essentially hackathons for data scientists — as well by facilitating longer-term engagements between volunteer data scientists or DataKind staff and organizations.

Saving money by proving what every landscaper knows

One of those volunteers is Brian Dalessandro, VP of data science for display advertising platform Media6Degrees. He met Porway at a data-industry function in New York in late 2012, was sold on DataKind’s vision (“[Jake's] very convincing that you should be passionate about it, too,” Dalessandro said) and got involved with his first DataDive shortly thereafter. The beneficiary organization: NYC Parks, which wanted help quantifying the benefits of tree pruning and the neighborhoods most at risk of tree damage from storms.

Delassandro tackling storm damage at the DataDive.

The benefits of mapping the neighborhoods in peril are pretty obvious, but doesn’t everyone already know that pruning keeps trees healthier and reduces the risk of falling limbs and other accidents? Kind of, Delassandro explained. Up to this point, all of the evidence has been anecdotal, which isn’t always enough when it comes to new expenditures in tight city budgets.

“They knew what they wanted to solve,” Dalessandro recalled, “they just didn’t know if they had the right ingredients to solve it.”

NYC Parks came to the DataDive with three datasets it hoped would do the trick — a census of every public tree in the city; a log of every work order on those trees; and a log of when each city block’s trees were pruned. After scraping some weather data and figuring out a working definition of “risk” that was both quantifiable and satisfied the department’s needs, Dalessandro and some others were able to solve the storm-prediction problem. Quantifying the effects of pruning turned out to be a hairier problem, though.

So, for the next four months, Dalessandro went to work during his spare time trying to solve it. Most of the work went to formatting the datasets so he could actually work with them like they were the same thing. This is actually a common issue with government agencies and non-profits, Porway noted, because they’re usually collecting data for accounting or reporting purposes rather than to use for statistic analysis.

Once the data was ready to go, though, Dalessandro was able to rework some existing code, which he had previously written to predict whether ads actually caused people to buy products, and do the actual analysis. “Instead of people converting, there’s trees and limbs falling off,” he analogized.

You know, classic parks department stuff. Source: Brian Delassandro

In the end, he found that pruning reduces hazardous work orders the following year on the blocks pruned by 22 percent. The next steps are to put his results into a business context, presumably to make a case for a better-planned and more-comprehensive pruning system. If it’s cheaper than sending out crews to fix damage, that’s probably not a bad idea.

Can you solve bigger problems without targeting a few ads?

As easy as it is to rip data science in the name of advertising, though, it seems like having that high-pressure business experience actually really helps with data volunteerism. One of SumAll’s missions is to teach the non-profits it works with to think about businesses in terms of what key performance indicators they want to track. Porway said DataKind is quite focused on teaching organizations to think like data scientists, even that just means structuring their data consistently so they can analyze it if they need to.

For his part, Dalessandro is excited to volunteer again, in part because he likes putting his well-honed technological skills to work in the name of the greater good. At previous jobs, he said, volunteering meant spending eight hours at the park pulling weeds or something equally mundane. However, he said, if someone needs a type of predictive model that he could build in his sleep, he could deliver truly meaningful results in just a couple hours.

If there’s a dark lining to this silver cloud, though, it’s that there will always be more problems than people to solve them. That doesn’t dissuade Porway, though, who sees a growing movement every time hundreds of people show up at a DataKind event, new chapters popping up overseas and the work being done by his peers in other organizations. Beside, he said, while some people are tackling difficult problems, there are lots of organizations who could benefit even from simple things like visualizations.

And free help is probably a better option than trying to bring those skills inside an organization. “Trying to hire data scientists to do this,” Porway said, “would be a Herculean task given how rare they are.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• How big data analytics drives competitive advantage
• Sector RoadMap: Social customer service in 2013

That’s why a group of researchers from Columbia University and Stanford University developed an algorithm that can do just that, turning those cells into something that resembles your social graph. This lets researchers see how the various cells are related to each other so they know , for example, where to focus cancer treatment and what to track as that treatment progresses.

The idea of representing large or complex data as a graph is nothing new, but it has taken on more prominence thanks to the rise of social media and those ubiquitous social graphs that map out who’s connected to whom. As we highlighted recently, however, graph analysis is becoming more popular outside the realm of social networks, and is being applied to problems that are more complex than just figuring out simple relationships within a network. In cases such as medical research, especially, graphs can provide a very effective way of seeing how potentially hundreds of thousands of data points spanning perhaps hundreds of variables are similar to each other.

That’s exactly what the team at Columbia and Stanford has done with a new algorithm that they’ve demonstrated within the realm of mass cytometry. According to a press release announcing the research (which is available via paid download at Nature Biotechnology):

“The method, called viSNE (visual interactive Stochastic Neighbor Embedding), is based on a sophisticated algorithm that translates high-dimensional data (e.g., a dataset that includes many different simultaneous measurements from single cells) into visual representations similar to two-dimensional ‘scatter plots’ ….

“The viSNE software can analyze measurements of dozens of molecular markers. In the two-dimensional maps that result, the distance between points represents the degree of similarity between single cells. The maps can reveal clearly defined groups of cells with distinct behaviors (e.g., drug resistance) even if they are only a tiny fraction of the total population. This should enable the design of ways to physically isolate and study these cell subpopulations in the laboratory.”

I assume they say similar to scatter plots because the algorithm is analyzing data across more than two dimensions, although the resulting chart is essentially the same (i.e., data points with similar characteristics will form clusters).

The results of viSNE, showing cell densities in diagnosis and relapse samples.

Whether or not they’re technically similar, this research seems similar to what Ayasdi is doing with its new data-analysis software based on a technique called topological data analysis. In both cases, though, the algorithms aren’t necessarily concerned with how data points interact with one another (like in network graphs), but rather what similar characteristics the points share. Ayasdi’s software has been used in cancer research, too, including on datasets spanning hundreds of patients and tens of thousands of variables.

In theory — although not likely in practice considering the complexity of the datasets medical researchers are dealing with — these approaches are similar to clustering approaches that are also popular among data scientists working with web companies. In areas such as e-commerce or email management, for example, where there isn’t a strong social element, companies can broadly break customers into distinct groups based on their behavior or interests.

A sample cluster of MailChimp subscribers.

Of course, curing cancer is a slightly more compelling — and difficult — goal than targeted advertising. The algorithms have to be precise so as not to miss similarities hidden within the mass of data. In the case of viSNE, the researchers say they’ve been able to spot small groups of cells (like 20 out of tens of thousands) that might be able to survive chemotherapy and increase the likelihood of a recurring tumor.

But we probably shouldn’t bee too quick to discount the work that web companies do as somehow less valuable than that of cancers researchers, for example. The big data era arguably started with the web, and web companies have generated some of the most important data-analysis techniques and technologies around today (see, for example, Google’s Jeff Dean, with whom I’ll be speaking at our Structure conference next month). As medical researchers start generating more and more data via cytometry, genome sequencing and even electronic medical records, it will be critical for individuals in all fields to keep track of what data scientists in other fields are doing and figure out how that might apply to their own work.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout

While there is something undeniably creepy about the Google Now service, I have to admit that it is also very useful — so much so that I couldn’t imagine going on a trip without it. I’m already imagining how it and other kinds of “anticipatory data” services (including Google News updates) might work through Google Glass.

Useful information when you need it

It’s not that Google Now is really all that revolutionary, in the sense of being surprising or magical or having whiz-bang special effects: it just collects a broad range of information about you and your activity from your search history, your calendar, your email, web services you are signed into, and so on, and then uses that to show you information that is relevant to what you are doing or where you happen to be (Google recently introduced it for iOS as well as Android).

In a way, that could be part of the reason Google Now is so appealing — it doesn’t try to impress you, it just works silently in the background, in more or less the way you would expect it to. That in itself is something to be grateful for.

The first time I noticed myself depending on it (or at least noticing how useful it was), came when I was getting ready for my flight to Italy: sliding upwards from the home button on the Nexus 4 showed a series of Google Now “cards,” and the first one said that my flight had been delayed by an hour. Since I was panicking at that point about how much I still had to do before leaving for the airport, that information was incredibly helpful. I could take a bit more time and relax.

Meanwhile, the second Google Now card showed the traffic on the highway and told me that I should probably give myself more time than usual to get to the airport — and when I got closer to the time of my departure, a third card showed my boarding pass information, including boarding time and the gate number (Google Now got that info from my calendar, but it also supports scannable boarding passes for a limited number of airlines).

Not revolutionary, but evolutionary

Again, none of this information was specific to Google Now, or derived magically by Google search trickery: I could have easily found out about my flight being delayed by using a service like FlightStats, or by checking the website for the airline or the airport itself — and I could have checked the traffic on any number of sites. But the point is that doing these things would take time, and I was already pressed for time. Seeing it all displayed in front of me in a simple way, without me having to do anything, was exactly the kind of thing a virtual assistant is good for.

Google Now continued to perform this kind of function while I was travelling (once I got a local SIM card, of course, so that I wouldn’t get robbed by my carrier for roaming charges). It told me that my connecting flight in Munich was on time, which allowed me to prepare for possibly not making my connection — and once I arrived in Italy, it informed me of the weather, the traffic from the airport in Rome, and also showed me photos of nearby sights that I might want to visit.

These latter aspects were also very useful for someone visiting a foreign country: I didn’t have much use for them while I was at home, but they instantly became much more important when I was travelling. Like the flight information or traffic, I could have found that content myself by doing a web search — but it was much handier to have it displayed for me automatically. And I started to imagine what it might be like to simply look at something like the Colosseum with Google Glass and have information about it appear in front of my eyes. Geeky? Yes. But also hugely useful.

The privacy tradeoff is worth it

The part that clearly disturbs some people about Google Now is the data collection that is involved in making it work: the tracking of your web searches, your calendar appointments, your location via GPS, the photos you have posted, the flights you are preparing to take, and so on. There’s no question that this is invasive — and some users will undoubtedly decide that it’s not worth the tradeoff, and choose to keep the information to themselves. I think the benefits outweigh the disadvantages.

Are there ways Google could use this information that I might not like? Of course there are. But I trust that Google is aware enough of the dangers — both legal and commercial — of engaging in that kind of behavior that they will avoid it. While some may choose to see Google’s ambitions in this area as evil, I think the company’s goal remains the same: to provide services that encourage users to spend more time on the internet and produce more data that improves Google’s search and/or advertising algorithms. And I am okay with that.

In return for providing some anonymized data and behavior patterns, I get access to a personalized assistant that is not only more unobtrusive than any human version would be, but is also faster and completely free. That’s a pretty good bargain.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

SkyPhrase is a very early-phase company that, according to its web site, has “made breakthroughs in algorithms that enable computers to understand more complex language with greater precision than has ever been possible.” The goal is to improve search functionality but also to give developers a new, easy way to incorporate natural language processing into their apps. The company was founded by Rensselaer Polytechnic Institute Professor Nick Cassimatis.

In January, MIT Technology Review reporter Rachel Metz covered the company and actually reviewed an early version of the technology as applied to searching through tweets and emails. It wasn’t yet trained to do what she wanted with tweets but, she wrote, did a “decent” job searching through emails. Part of what makes it work appears to be its ability to understand conjunctions, even if it doesn’t yet have semantic capabilities: “I could search for, say, ‘e-mails from Bob Loblaw in December and January about recipes with a PDF,’ or ‘e-mails from Bob Loblaw or Tobias Funke about cookies in December,’” Metz explained.

Nanostraws in a cell

Breakout Labs’ other new investment, Stealth Biosciences, is a team of Stanford professors, executives and entrepreneurs that has invented a way to get materials into and out of individual cells and to monitor their activity via electric probe. Called Nanostraws and Stealth Electrodes, respectively, the companies two techniques do just what they sound like they do: NanoStraws let doctors inject or extract material from cells in the aims of advancing research and delivering personalized medicine, while the electrodes “automate long-term intracellular electrical recordings of neurons and heart cells.”

Stealth Biosciences, in particular, seems like a heady endeavor, but that’s exactly what Breakout Labs is all about. Launched in 2011, the organization aims to fund projects too early in their lives to attract traditional venture capital. Those funded aren’t giving up large equity stakes in their companies, but are expected to provide a “modest portion” of their revenues back into the program to fund the next generation of Breakout Labs investments. Other investments thus far include Modern meadow — a company trying to create artificial meat using 3-D printers — and AVEtec, a Canadian startup trying to harness the power of tornadoes for good.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Social customer service in 2013
• How emerging technologies will influence collaboration
• NewNet Q4: Platform mania and social commerce shakeout

Aiming to figure out why America is becoming more politically polarized despite traditional beliefs that societies naturally move toward the middle, a group of Stanford researchers considered how our natural biases affect the way we interpret information. What they found is that people tend to view the world through red- or blue-colored glasses: when we see inconclusive information, we intepret it in ways that support our natural political biases and ignore the aspects that don’t. So if you show the exact same piece of inconclusive information to a group of people, it will likely lead to more polarization rather than to general consensus on the meaning.

It turns out, this phenomenon extends beyond clearly biased media such as FOX or MSNBC and into more objective content sources on the web. When the researchers applied their model to online recommendation engines, they found that pieces of content most-relevant to users are “always polarizing,” whereas pieces of information that are merely similar to something someone already likes are only polarizing if the person is already biased. In short: While they’re able to ignore or at least view objectively less-important stuff, even pretty middle-of-the-road people will take a hard stance on stuff that matters to them.

Of course, how one reacts to research like this largely depends on what one is trying to accomplish. The researchers involved appear to be all about moving people toward the middle on some issues, which is why they created a federal-budget app called Widescope that lets people configure their own budgets and then shows them the similarities with the various budget proposals floating around Washington, D.C. They’ve also looked into creating social systems that counteract polarization by using trusted information sources (a press release explaining the research suggests Rush Limbaugh or Rachel Maddow) to present information that biased individuals might otherwise be inclined to dismiss.

Applied generally to the web, this approach might help mitigate some of the effects of the hyper-personalized experience that’s now possible. You know, the kind of thing that happens when you fill up RSS readers with sources you like, follow like-minded people on Twitter, and  sign up for services that use machine learning to surface even more of the same content based on that homogeneous reading activity. Or when you keep searching for the same stuff on Amazon or viewing the same types of movies on Netflix.

Services that go beyond “injecting serendipity” into their content feeds could actually try to broaden users’ minds by surfacing content that’s in some ways very different or counterintutive to what a simple interest graph might show. I’m not sure how this would look algorithmically, but I’m envisioning, for example, a semi-regular insertion of content from sources or genres considered the opposite of a readers’ norms but that touch upon topics they’re interested in. Or vice versa.

I genuinely believe most web startups trying to tackle the problem of content curation want to be helpful as possible, are aware of issues such as biased assimilation and are at least considering methods for counteracting it in order to give users a broader view beyond just what those users think they want to see.

On the other hand, if you wanna lock people into their current beliefs or their current content-consumption habits, that’s probably a lot easier to do. And sadly, for some politicians and special interest groups, that probably suits them just fine.

Feature image courtesy of Shutterstock user Kutlayev Dmitry.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How energy data will impact the smart grid
• Over the top in 2012: trends and technologies to watch
• The importance of putting the U and I in visualization

The Netflix way of processing data online, offline and nearline.

The whole point of its data architecture is to tackle latency by pointing workloads and tasks toward systems designed to work at their speed. People love to think about Hadoop when they think about web data, but the reality is that relying solely on batch processing means data can get stale and applications probably don’t include the newest user input.

Netflix uses online processing for receiving information from users in real time and serving up responses right away, such as looking at a new rating or some other customer action to change the set of movies shown to the customer. Real-time processing works best when algorithms are relatively simple and when data is on the smaller side. The data feeding in to computations must also be available right away.

Nearline processing happens when the data needs to be computed in real time but can be stored for serving up at a later point in time. This option makes sense when computations are more complex and are amenable to a more-traditional database-oriented approach. Netflix uses a variety of databases, including MySQL, the NoSQL Cassandra database and its own homemade EVcache system.

Offline processing in Netflix’s world might also be called batch processing — think bigger and longer-term Hadoop jobs. It also fits for compute-heavy projects to train new models that will come into use at a later date. And it’s a backup for situations when real-time processing isn’t possible.

This online-nearline-offline approach is fairly common among web companies that understand that different applications can tolerate different latencies. LinkedIn has built its data infrastructure with the same general theory in mind. Facebook, too, has thought deeply about this. The social network recently detailed a new memcached-like data store called McDipper that foregoes DRAM for flash in order to cut costs for tasks that can live with slightly higher latency.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

The researchers — who come from the University of Southern California, Scripps Clinic, the Scripps Research Institute, the University of California, San Diego Moores Cancer Center and Memorial Sloan-Kettering — combined autopsy data from 163 cancer cases (all from before the advent of radiation therapy in order to analyze the natural spread) with applied mathematics in order to carry out their study. What they found, according to a press release about the research is that

metastatic lung cancer does not progress in a single direction from primary tumor site to distant locations, which has been the traditional medical view. Instead … cancer cell movement around the body likely occurs in more than one direction at a time.

How cancer cells spread. Source: PLOS One

Moreover, they found certain organs tend to spread cancer cells more aggressively, while others tend to act as sponges for cancer cells. These sponge organs might still grow tumors, they just don’t disperse the cells.

The PageRank analogy

The mathematics involved here — called Markov chain models — are similar to what Google uses to determine what web pages are the highest-quality for any given search query. Only whereas Google uses the number and quality of links to determine the probability of a web surfer landing on any given page, these researchers are trying to predict the PageRank of tumors, if you will. So, generally speaking, a kidney would likely have a higher PageRank than a liver because the kidney is more likely to spread cancer cells throughout the body (or, in web-search terms, generate a lot of links to itself).

The network path of cancer cells from lung to liver. Source: PLOS One

As data volumes proliferate and relationships between data points become more complex, Markov models are actually becoming pretty popular. Netflix uses them in order to predict the movies users will want to watch next.

The weighted connections between various states or web pages or whatever someone is ranking are often expressed as the nodes and edges of a graph. Graphs, of course, have become part of the everyday web lexicon thanks to the various social graphs and interest graphs that analyze who we’re connected to (and how) and the types of topics we browse online.

The web as a data science proving ground

So in the end, perhaps, the most-important contribution of the worldwide web won’t be the revolution in terms of how we access information, but the web’s function as a proving ground for advanced statistical methods starring very large and complex data sets like those found in the medical world. Already, for example, another group of medical researchers has used a Markov variant in order to create a model they think can prescribe better treatment plans because it analyzes the costs and patient outcomes usually associated with a given treatment for a given symptom.

Tracking a cholera outbreak across a river network. Source: Physical Review Letters

Last year, a group of Swiss researchers developed an algorithm that, having access to a relatively small amount of data, can track anything from Twitter rumors to disease outbreaks back to their source. A company called Syapse uses the graph structure to chart the relationships among words across different medical specialties.

One would also be remiss in ignoring the computing and data-storage innovation spurred by the web that has improved our ability to handle massive amounts of genetic and other data. As the lung cancer researchers explain in their paper:

One of the strengths of such a statistical approach is that we need not offer specific biomechanical, genetic, or biochemical reasons for the spread from one site to another, those reasons presumably will become available through more research on the interactions between CTCs and their microenvironment. We [have created] a quantitative and computational framework for the seed-and-soil hypothesis as an ensemble based first step, [that] then can be further refined primarily by using larger, better, and more targeted databases such as ones that focus on specific genotypes or phenotypes, or by more refined modeling of the correlations between the trapping of a CTC at a specific site, and the probability of secondary tumor growth at that location.

The long story short is that the more data we have and the easier we can analyze and map it, the better we can treat — and perhaps even cure — cancer and other complicated diseases.

Feature image is a network map of how lung cancer spreads between organs, where each numbered node correlates with a specific organ.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• The importance of putting the U and I in visualization

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• Ups and downs for cleantech in Q1

The premise is surprising, given that many database companies make a point of divvying up the responsibility of processing to keep latency low. But if cloud providers build on and adopt the researchers’ DBSeer algorithm, it could improve cloud database performance.

Infrastructure-as-a-Service (IaaS) providers run virtual machines on servers. That might not be the most efficient approach for databases, because resources aren’t shared among the applications running on any given server, the researchers argued in a recent paper. It might be better to observe current workloads, predict the needs of future workloads and bring together the different sorts of loads on different servers. Then cloud providers could adjust service-level agreements to promise a certain level of latency rather than charge customers based on the number and size of virtual machines, the researchers noted.

DBSeer might also be of interest to database appliance and server vendors. Teradata is incorporating the algorithm into proprietary software. Meanwhile, one of the MIT researchers, Carlo Curino, now works at Microsoft, and Chinese webscale server vendor Quanta funded the research.

So far, DBSeer, which is available on GitHub, has only been shown to accurately predict workload needs for transactional MySQL databases. More research would be necessary to apply the algorithm to other database management systems.

The change in thinking could make good financial sense. The more hardware in use inside a cloud provider’s data centers, the more expensive it is for customers. If the appliances could work more efficiently, costs could drop.

Feature image courtesy of Shutterstock user Johann Helgason.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• How fourth-quarter 2012 will affect IT spending in 2013
• The role of converged infrastructure in the data center

We all know that big data has so much promise. For a very large number of problems today, the effective use of data is a bottleneck. The drug discovery problem is more about data than chemistry. The discovery of new energy sources is more about data than geology. It’s the same for tracking terrorists, detecting fraud, and more.

Today, we recognize that these, and many other critical global issues, are all data problems. This fact alone has given rise to a huge investment into big data, created the hottest job title around –data scientist — and propelled the valuations of private data analytics providers into the billions. However, imagine the endless possibilities when the world is operating on the insights gathered from 100 percent of its data.

Realizations

Where do you start when you have a data set as large as the human genome, for example, or President Obama’s recent call to map the human brain? To achieve the breakthroughs we need to address the world’s most perplexing problems, we need to fundamentally change the way we gain knowledge from data. Here’s what we need to start thinking about:

• Starting with queries is a dead end: Queries are not inherently bad. In fact, they are essential once you know what question to ask. That’s the key: the flaw is starting with queries in the hope that they will uncover a needle in the massive digital haystack. (Spoiler alert: they won’t.)
• Data has a cost: Storing data is no longer expensive, in most cases. Even querying large amounts of data is becoming more cost effective with tools like Hadoop and Amazon’s Redshift. This is just the hard cost side of the equation, though.
• Insights are value: The only reason why we bear the cost is because we believe that data has insights that unlock value. Ultimately, the undiscovered insights that organizations miss have a much higher cost in terms of being able to solve big problems quickly, accelerate innovation and drive growth. The cost of data collection can be high, but the cost of ineffectual analysis is even higher. The tools for getting at insights don’t exist today. Today, we rely on very smart human beings to come up hypotheses and use our tools to validate — or invalidate — those hypotheses. This is a flawed strategy since it relies on (arguably smart) guesswork.
• You have the right data today: There’s often a belief that, “If we only had more data, we could get the answer we’re looking for.” Far too much time and money is wasted collecting new data when more can be done with the data already at hand. For example, Ayasdi recently published a study in Nature Scientific Reports that shows important new insights from a 12-year-old breast cancer study that had been thoroughly analyzed for over a decade.

Big data is the beginning, not the end

I’m very concerned that the growing hype around the term big data has set us all up for disappointment. Query-based analysis is fine for a certain class of problems, but it will never be able to deliver on the expectations the market has for big data.

We are on the cusp of critical breakthroughs in cancer research, energy exploration, drug discovery, financial fraud detection and more. It would be a crime if the passion, interest and dollars invested to solve critical global problems like these were sidetracked by a “big data bubble.”

We can and should expect more from data analysis, and we need to recognize the capabilities that the next generation of solutions must be able to deliver:

• Empower domain experts: The world cannot produce data scientists fast enough to scale to the size of the problem set. Let’s stop developing tools just for them. Instead, we need to develop tools for the business users: biologists, geologists, security analysts and the like. They understand the context of the business problem better than anyone, but might not be up to date with the latest in technology or mathematics.
• Accelerate discovery: We need to get to critical insights faster. The promise of big data is to “operate at the speed of thought.” It turns out that the speed of thought is not that fast. If we depend on this approach, then we will never get to the critical insights quickly enough because we’ll never be able to ask all of the questions of all of the data.
• Marriage of man and machine: To get to those insights faster, we need to invest in machine intelligence. We need machines to do more of the heavy lifting when it comes to finding the clusters, connections and relationships between data points that gives business users a much better starting point to begin discovering insights. In fact, algorithmic discovery approaches can solve these problems by looking for rare, but statistically significant signals in large datasets that humans would never be able to find. For example, in a recent study, previously unreported drug side effects were found by algorithmically searching through web search engine logs.
• Analyze data in all its forms: It’s understood that researchers need to analyze both structured and unstructured data. We need to recognize the diversity and depth of unstructured data: text in all languages, voice, video and facial recognition.

When it comes to the evolution of big data, we’ve only begun to scratch the surface. It stands to reason that if we continue to analyze 1 percent of data, then we’ll only tap into 1 percent of it’s potential. If we’re able to analyze the other 99 percent, then think about all of the ways that we can change the world. We can accelerate economic growth, cure cancer and other diseases, reduce the risk of terrorist attacks, and many other big ticket challenges that we’re faced with.

That’s something that we can all rally around.

Gurjeet Singht is the co-founder and CEO of Ayasdi, an insight discovery platform built on topological data analysis technology. He will be speaking at Structure: Data, March 20-21 in New York.

Have an idea for a post you’d like to contribute to GigaOm? Click here for our guidelines and contact info.

Feature image courtesy of Shutterstock user Sergey Lavrentev.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout