What seems to be missing from all of these discussions, though, is a dialogue about how to steer around this bottleneck and make big data directly accessible to business leaders. We have done it before in the software industry, and we can do it again.

To accomplish this goal, it’s helpful to understand the data scientist’s role in big data. Currently, big data is a melting pot of distributed data architectures and tools like Hadoop, NoSQL, Hive and R. In this highly technical environment, data scientists serve as the gatekeepers and mediators between these systems and the people who run the business – the domain experts.

While difficult to generalize, there are three main roles served by the data scientist: data architecture, machine learning, and analytics. While these roles are important, the fact is that not every company actually needs a highly specialized data team of the sort you’d find at Google or Facebook. The solution then lies in creating fit-to-purpose products and solutions that abstract away as much of the technical complexity as possible, so that the power of big data can be put into the hands of business users.

By way of example, think back to the web content management revolution at the turn of the century. Websites were all the rage, but the domain experts were continually banging their heads against the wall – we had an IT bottleneck. Every new piece of content had to be scheduled and sometimes hard-coded by the IT elite. So how was it resolved? We generalized and abstracted the basic needs into web content management systems and made them easy for non-techies to use. As long as you didn’t need anything too crazy, the problem was solved easily, and the bottleneck averted.

Let’s dig a little deeper into the three main roles of today’s data scientist, using online commerce as a backdrop.

Data Architecture

The key to reducing complexity is to limit scope. Nearly every ecommerce business is interested in capturing user behavior – engagements, purchases, offline transactions and social data – and almost every one of them has a catalog and customer profiles.

Limiting scope to this basic functionality would allow us to create templates for the standard data inputs, making both data capture and connecting the pipes much simpler. We’d also need to find meaningful ways to package the different data architectures and tools, which currently include Hadoop, Hbase, Hive, Pig, Cassandra and Mahout. These packages should be fit for purpose. It comes down to the 80/20 rule: 80 percent of big data use cases (which is all most ecommerce businesses need), can be achieved with 20 percent of the effort and technology.

Machine Learning

Surely we need data scientists in machine learning, right? Well, if you have very customized needs, perhaps. But most of the standard challenges that require big data, like recommendation engines and personalization systems, can be abstracted out. For example, a large part of the job of a data scientist is crafting “features,” which are meaningful combinations of input data that make machine learning effective. As much as we’d like to think that all data scientists have to do is plug data into the machine and hit “go,” the reality is people need to help the machine by giving it useful ways of looking at the world.

On a per domain basis, however, feature creation could be templatized, too. Every commerce site has a notion of buy flow and user segmentation, for example. What if domain experts could directly encode their ideas and representations of their domains into the system, bypassing the data scientists as middleman and translator?

Analytics

It’s never easy to automatically surface the most valuable insights from data. There are ways to provide domain-specific lenses, however, that allow business experts to experiment – much like a data scientist. This seems to be the easiest problem to solve, as there are a variety of domain-specific analytics products already on the market.

But these products are still more constrained and less accessible to domain experts than they could be. There is definitely room for a friendlier interface. We also need to take into consideration how the machine learns from the results that analytics deliver. This is the critical feedback loop, and business experts want to provide modifications into that loop. This is another opportunity to provide a templatized interface.

As we learned in the CMS space, these solutions won’t solve every problem every time. But applying a technology solution to the broader set of data issues will relieve the data scientist bottleneck. Once domain experts are able to work directly with machine learning systems, we may enter a new age of big data where we learn from each other. Maybe then, big data will actually solve more problems than it creates.

Scott Brave is co-founder and CTO of Baynote, an e-tail and e-commerce advisory business. He is also an editor of the “International Journal of Human-Computer Studies” (Amsterdam: Elsevier) and co-author of “Wired for speech: How voice activates and advances the human-computer relationship” (Cambridge, MA: MIT Press).

Photo courtesy of Sergey Nivens/Shutterstock.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms
• Infrastructure Q1: Cloud and big data woo enterprises

At its core, oxdata’s flagship product, called H2O, is a statistical analysis engine that uses the Hadoop Distributed File System (HDFS) as its storage platform, but the goal is to make it as simple as using a Google service such as BigQuery. Users will interact with H2O via a simple web-search-like bar and standard R statistical-analysis syntax, but H2O will run machine-learning algorithms behind the scenes. Alternatively, users can call out to H2O from Microsoft Excel or the RStudio integrated development environment using a REST API.

Although BigQuery is a SQL service hosted by Google, 0xdata follows a similar theory on simplicity.

However they choose to leverage the product, Ambati said, the scale of the underlying data and the complexity of running advanced analysis are details that need to be hidden. It’s the same theory that underlies Platfora, the company Ambati co-founded last year with his former DataStax colleague Ben Werther, although their approaches appear to be different. Whereas Platfora is trying to disrupt the data warehouse market by building a next-generation user experience atop Hadoop, 0xdata is trying to change the way users interact with popular statistical software such as R.

But either way, Ambati says of new data-analysis products, “[There are] no bragging rights for making it simple. If you don’t do that, you won’t be able to go forward.”

oxdata is also putting a focus on speed, both in terms of how fast it processes data and how fast it lets users react. Google search changed our thinking around how many questions people can ask successively, Ambati explained, and data analysts should have the same experience. That’s why H2O provides approximate results at every step in the analysis process. Rather than wait for the entire job to run and the exact results to be computed, users can get a general idea of results and kill the job and start over quicker if they’re completely outside the expected range.

But it will be a while before the public gets a chance to see whether H2O lives up to its promises. Ambati said the product is just four months into development and won’t have its first set of algorithms available for another few months. His team of eight engineers has “built a lot of cool stuff,” but now it needs to round out the process and turn its code for H2O into an actual product.

Still, having decided to tackle data as a system, Ambati and his team are having a lot of fun. “We are live-and-die-with-infrastructure people,” he said, but for a bunch of folks who spent a lot of time learning math, it’s like going back to the their days as computer science students.

Feature image courtesy of Shutterstock user Bruce Rolff.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How data warehousing is now a cost-effective solution for businesses
• How to manage big data without breaking the bank

Ted Corbett, the director of knowledge management at Seattle Children’s Hospital, is using software from a company called Tableau to draw smart inferences from the 10 terabytes of data locked up in his servers and warehousing appliances. The hospital, which employs 5,000 people, uses the visualizations and easy access to data hidden away in multiple places to cut down on waste, reduce errors in medicine and help plan clinical trials. As organizations seek to store, analyze and derive insights from their data, companies are creating software to help them make sense of it all — because it’s not how big the data is, but what you can do with it.

Digestible data 

Tableau is one of several companies attempting to funnel massive amounts of data into a more human understanding. Others include Karmasphere, Revolution Analytics,TIBCO and SAS. Corbett explains that offering employees access to data via a simplified dashboard has helped the hospital better schedule its time in operating rooms and eliminate waste from the supply chain by improving the information needed for the hospital to implement some of its lean manufacturing tenets.

“Nurses and doctors, when they need test tubes or syringes, they would hoard things so when it got busy they would have them, but knowing whether you are pulling enough of the right supplies and having them available is a way to save on costs. So far we’ve saved $3 million out of the supply chain, and using Tableau we can find new ways to eke out more savings,” Corbett said.

Tapping data for real insights

Visualizations are one way companies are mining their existing data. Other companies are making products that integrate into existing ways of managing data such as Karmasphere or even Microsoft with its data marketplace product, where a manager or analyst can buy access to data and import it into Excel. Instead of creating more frivolous infographics, these products are able to help businesses tap into existing data better and make some kind of meaning from it.

These companies may not have huge adoption today, but as the amount of data companies analyze expands, such solutions should become more popular. As Corbett notes he’s expecting to see up to a tenfold increase in data over the next few years thanks to things such as M2M communications within the hospital, personal genomics in medicines and electronic health records. “We’ll have an expansion in machine data — everything is Wi-Fi-enabled and pumping data all around the hospital– and there will be genomic data and electronic records,” Corbett said. He estimates that each patient record contains a “couple of gigs of data per patient,” which can add up.

Data tools for the masses

It’s helpful to have tools that researchers and statisticians can use to make sense of the massive piles of data they have to sift through. But for big data to really become a game-changing force in business, companies will have to develop tools for the common man — or at least the middle manager. Much like broadband, computers, electricity and other big changes in productivity, it has to filter down to the masses to really change the world.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout

As much as I’m in favor of anything that improves the chances of finding true love (or any kind of love, for that matter), what fascinates me the most about OkCupid is OkTrends, the company’s official blog.

OkTrends Brings Big Data to the Masses
Written largely by Christian Rudder, the company’s co-founder and editorial director, OkTrends is a veritable treasure chest of sexy social insight generated through a highly creative mash-up of off-the-shelf and DIY analytic techniques.

A recent post, “Big Lies People Tell in Online Dating,” reveals some pretty amazing truths behind the tall tales that are told regularly in the quest for romance. With unflinching mathematical precision, accompanied by several convincing charts, the post shows how people routinely lie about their height, income and looks. This revelation isn’t exactly surprising, but it’s fascinating to see how Rudder and his merry crew of quants use the power of analytics to strip away the mythology and tell the real story.

One of the most interesting aspects of the blog is the size of the samples. As Rudder noted in the blog’s inaugural post back in June 2009:

… a word about statistical validity: the best questions on OkCupid have been answered over a million times. Therefore we have unique insights into the American mindset. A quick comparison:

Old media could only get 3,050 people to answer a poll about Obama. And it was enough to call the election with confidence. OkCupid, on the other hand, can ask the world’s most personal questions and get hundreds of thousands of answers.

Open Source Is the Key to Deeper Insights
The analytic techniques used to crunch the numbers and surface the patterns tend to vary. In the early days, when OkCupid had fewer members and the data sets were smaller, Excel sufficed. But as the site’s membership rapidly grew into the millions, it was not uncommon for surveys to generate responses from 500,000 members. It became clear to Rudder that Excel by itself could not handle data from 500,000 responses; a more robust solution was required. Recently, OkCupid added statistical packages written in R to its mix of analytic tools.

R is an open-source language designed specifically for statistical analysis (see disclosure below). Unlike some of the more widely used, proprietary, analytic tools, programs written in R can handle the larger and more complex data sets generated by OkCupid’s growing base of users. That makes R a good choice for data scientists who are interested in pushing the envelope of traditional statistical analysis.

“R helps us get a quick overview of the data, which can save us a tremendous amount of time,” says Rudder. “If we had to do everything in Excel, it would take forever.”

Rudder’s crew uses R to visualize big data quickly, something they couldn’t do with Excel. “R lets us get a ‘zoomed-out’ view of what’s going on with the data, which helps us decide quickly if the tack we’re taking with the data is yielding something interesting. Once we figure out what we’re looking for, and we start narrowing our focus, then we can move into Excel,” says Rudder.

Rudder has also benefitted from the open-source community. “If I have some data and I have an idea for analyzing that data, the chances are good that someone in the R community has already written a program that does what I need,” says Max Shron, a data scientist at OkCupid. “That’s the nice thing about open source. People just go ahead and write programs. It’s a tremendous time saver.”

Shron and Rudder recently used R packages to analyze data for a study comparing gay and straight dating habits. “There was a tremendous amount of data,” says Rudder. “When we were looking at things like sex partners and messaging, R was very helpful. We could see the patterns very quickly and we knew we had something to write about.”

The low cost of open-source software is also a factor. “If we had to buy a package from a major vendor — even just one license for Max — we’d blow our budget for software. It’s much easier just doing some of this stuff in R,” says Rudder.
R is not a panacea for solving every data challenge – at least not yet, says Rudder. “I am a long-term Excel user; I’ve been using it since I was 12. So when it comes down to generating the final graphics, I usually like Excel. But I feel that with the right amount of work, R could probably do anything.”

OkTrends shows how big data and analytic science can hold a mirror to society, revealing its strengths and weaknesses. It’s gratifying to know that many of the advanced analytic processes running “under the hood” at OkTrends are hand-crafted from open-source components created by a worldwide community of people writing code for love, not money.

For more insights from the big data landscape, come to GigaOM’s Structure: Big Data conference on March 23 in New York City.

Mike Minelli is an executive VP at Revolution Analytics, a company founded in 2007 to foster R analytics by creating programs to make it easier for data scientists to analyze large amounts of data. Note: Neither Minelli nor Revolution Analytics has a business relationship with OkCupid.

Related content from GigaOM Pro (subscription req’d):

• Big Data 2011 Preview
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• Report: Health Care’s Climb To the Cloud

In the video below, he talked about why we need to speak to our data and how that conversation can change the way we innovate and do business.

He also covers the changes that have taken place in his 40-odd years of working with data. Essentially, the introduction of scalable architectures such as Hadoop and cheaper computing have enabled people to look at data in real time. Combine that with a means of analyzing it, such as Revolution R, and it’s possible to offer people who aren’t statisticians a way to interact with big data and use that information to inform their daily decision-making. For the full story, watch the video, and make plans to come see Lee Edlefsen, chief scientist at Revolution Analytics, at our Structure Big Data conference taking place on March 23 in New York City.

Related content from GigaOM Pro (subscription req’d):

• Big Data 2011 Preview
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• Report: Health Care’s Climb To the Cloud