Blind trust in black box, or click-and-run, software is a growing problem in science, according to a commentary published Thursday in the journal Science, and the concern extends beyond formal research to other domains that use high performance computing.

The researchers who addressed the “troubling trend in scientific software use” were motivated by a growing unease that the abundance of powerful software is letting scientists derive answers without a thorough understanding of what the software is doing. Software snafus have been responsible for some high-profile data misinterpretations and retractions.

This wouldn’t normally cause a blip on the average citizen’s radar, but now a lot of these scientific conclusions have real-world implications, from climate modeling and weather forecasting to high volume financial trading. In any domain using big data, misplaced trust in the power of software can be problematic, particularly when the decision makers don’t know what the software they are using is doing, said lead author Lucas Joppa of Microsoft Research.

So what does ecology have to do with any of this? Joppa is an ecologist by training, and works on computational techniques in that field that may also have applications for big data more broadly. He and his colleagues surveyed scientists in a sub-field of ecology — species distribution modeling (SDM) — to find out how they choose software and how well they understand its inner workings.

“Lots of SDM techniques are only available as computational methods, but there is a lot of discourse going on in the literature about whether the methods themselves are correct,” said Joppa. Scientists use SDM to forecast where plants and animals will be in the future given current numbers, known habitats, and climate change. It’s a niche area of research, but the disquieting survey results should be noted in any domain where forecasting is done by plugging data into software.

Only 8 percent of the more than 400 scientists who responded had validated their modeling software against other methods. “The number speaks for itself,” said Joppa. “The real crux of the problem is the results from software being published in a peer-reviewed journal, versus the software itself having been peer-reviewed,” which is rare. Software packages, whether proprietary or not, are often black box systems that can’t be opened and inspected. Even if you can get under the proverbial hood, like with open source software, said Joppa, most people will still have no idea what they are looking at, or how to judge its quality.

To top it all off, having confidence in what your software is doing results in a massive computational catch-22: how do you know the software is giving you the right answer, if you can’t get the answer without running the software? The level of confusion over what algorithms are doing in the SDM field is illustrated by a debate over which of two statistical techniques is superior. It turns out, Joppa explained, that the two techniques were mathematically equivalent, but the ways they were implemented in software resulted in big predictive differences.

This sort of mix-up isn’t surprising given the messy nature of software development (if you can even call it that) in research environments. Joppa lauded efforts like Software Carpentry that teach scientists basic software fundamentals for better programming, and said the days of getting a doctorate by merely pushing a button are over.

“Scientists themselves can learn a bare minimum of software engineering,” said Joppa. On the flip side, he said computer science students should have more exposure to scientific methods. “People with traditional software engineering training become uncomfortable with the way scientists want to work with software, where the design and specs are constantly changing. The way that scientific software is built is fundamentally different from consumer apps.”

Developers of scientific software, like MathWorks or SAS, may want to watch this space. If Joppa’s suggestions are implemented, journals may start requiring that even proprietary software be opened up for inspection and peer-review. Nearly half of the surveyed ecologists report using free statistical language R as their primary software, so maybe there is hope yet, both for open, inspectable code, and for computational science becoming more accessible while yielding trustworthy, high impact results.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Health care and big data in 2012
• The importance of putting the U and I in visualization
• Pervasive Software retools for cloud, big data: will it be heard?

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

The way it works, said Mortar Co-founder and CEO K Young, is that his company will choose eight companies (in addition to the two it has been working with already) to implement custom systems based on their specific needs and businesses. Mason, Conway and Shron will split their time among the 10 total companies, but will be much more than advisers — they’ll actually dig into the data and work hands-on to ensure the right techniques and algorithms are applied in the right places.

The applicant companies will keep any custom code, but the ultimate goal from Mortar’s perspective is to learn some best practices and create reusable building blocks that will let anyone create recommendation engines without pre-existing data science knowledge. Recommendation engines are commonplace on large web sites (Netflix, Spotify, iTunes, Google, Amazon, LinkedIn, Eventbrite and the list goes on) but smaller companies can sometimes struggle to do them, or to do them well. Young hopes Mortar can establish an open source reference architecture of sorts that makes it easy to implement everything from building data pipelines to the actual algorithms that power recommendations.

“They’re really common and they’re really useful, but they’re really hard,” he said. “That’s why [a reference implementation] hasn’t been done before.”

They can get pretty complex, as evidence by this Netflix example.

Presently, Young explained, anyone wanting to build a recommendation system probably knows some of the algorithms to begin with and then gets to work researching how to implement them with specific processing frameworks (e.g., MapReduce) and on their specific data. Alternatively, they might have to hire a consultant that helps them build the recommendation engine. Either way, he noted, they’re probably not open sourcing it at the end because it’s presumed too valuable a competitive edge.

Mortar Data’s recommendation framework will be based on Pig, Python and Java, just like the company’s flagship platform for creating Hadoop jobs. Those languages will make the implementation more accessible and customizable by more people, Young said.

Really, he added, any web site or service that has multiple customers and deals with multiple entities — be they restaurants, songs, dating profiles, artisan necklaces, what have you — should have some sort of recommendation engine to help provide a more-intelligent customer experience. “It should become so ubiquitous that any service you go to knows enough about you to put forward the things you actually want to see,” Young said.

There is, however, one catch to Mortar’s plans as they stand: Because the service is hosted on Amazon Web Services, anyone interested in having Mason, Conway, Shron and Mortar work on their systems must have their data in AWS or be able to move it there. The initial reference implementation will likely be AWS-centric, too, but Young hopes contributors will use it and share methods for running it atop other platforms.

Feature image of Hilary Mason at Structure: Data 2011 courtesy of Pinar Ozger (www.pinarozger.com).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises

During one panel, the assertion that data science is dead was indeed debated. Along with the habitual tension between end user requirements for businesses and consumers and the “elitist” ideas of data scientists and engineers, other themes explored included increasing accessibility to data, as well as changing behaviors and encouraging better decision-making with data. Everyone from sales and marketing people to fitness enthusiasts, it turns out, can be motivated by pretty pictures.

As IBM’s Alah Keahey put it during a panel, “there is a hunger for friendly data,” and visualization can help to humanize those threatening terabytes. Here are a selection of new, and new-to-us, visualization tools that came up at the meeting.

Bringing climate change home: Databasin.org

A mapping and analytics platform from the Conservation Biology Institute that has 10,000 datasets on everything you need to understand how extreme weather will impact natural resources, renewable energy, and endangered species. Here is one projection of maximum temperatures in 2080.

Sparkvis by Chloe Fan

This app is for the quantified self junkie who loves to interpret their burned calories as abstract art. The research behind the colorful display of Fitbit (see disclosure) data is explained here. Image via QuantifiedSelf.com

Disqus Gravity

The commenting platform’s diverse content is brought together in an interactive and live visualization. Pulling from about 500 sites that use Disqus, Gravity brings together the “small” data of individual comments within the context of 11 content categories. Another visualization, Orbital, shows realtime comments geolocated on a spinning globe.

IBM Many Eyes

Originally conceived by visualization guru Martin Wattenberg and colleagues in 2007, Many Eyes lets you plug in any dataset and generate nifty figures. Here, for example, is the distribution of U.S. foreign aid over a 60-year period.

Disclosure: Fitbit is backed by True Ventures, a venture capital firm that is an investor in the parent company of GigaOM. Om Malik, founder of GigaOM, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

Machine learning is such a hot topic right now because data volumes are becoming so large and complex that humans alone can’t query their ways through them fast enough or intelligently enough to spot latent patterns among the mess of data. It’s the algorithmic engine that powers a bunch of Google services and your Netflix recommendations, as well as web content-curation service Prismatic and alternative-underwriting platform ZestFinance. As we covered in some detail at this year’s Structure: Data conference, machine learning is particularly powerful when its ability to correlate tens of thousands of variables is paired with human judgment about what really matters.

Skytree co-founder Alexander Gray (second from left) at Structure: Data 2012. (c) Pinar Ozger

Skytree, for its part, sells a product called Skytree Server that lets users run a wide variety of machine learning algorithms across whatever data they have. It might be an oversimplification, but Skytree is essentially a souped-up version of statistical-analysis packages like SPSS or SAS that’s designed to run fast — and, more importantly — without sampling across a scale-out server architecture. In March, the company also rolled out the beta version of a new product called Adviser that can run on a laptop and walks more-novice users through the analysis of their data, including what methods were used and why, and whether the findings are statistically significant.

I suspect we’re just seeing the opening salvo in what will be a rush to fund machine learning startups over the next couple of years. Skytree is among a number of increasingly promising startups in the space, including (but certainly not limited to) Ayasdi and Quid. As more individuals see the promise of machine learning and get skilled in applying it to their particular problems and datasets — as UPS apparently has — it could become become one of the go-to analytic methods in the big data era.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Health care and big data in 2012
• Takeaways from the second quarter in cloud and data

Obviously, data is nothing new to the automotive industry — companies have been trying to optimize supply chains and analyze sales numbers for decades — but the advent of big data, as well as related technlogies such as sensors and smartphones, is changing how companies are thinking about data. Ford isn’t alone in its quest to take advantage of these new technologies, either. For example, General Motors collects data from its OnStar system to help lower drivers’ insurance premiums, and also collects lots of data on its Chevrolet Volt electric car that it feeds to drivers via a mobile app. We recently noted how a luxury automobile company used big data software from Aster Data Systems to determine the relationships between malfunctions so it could provide a more thorough and beneficial service-department experience.

But in an industry notoriously unwilling to talk about information technology, Ford’s experiences might shed a lot on what other companies are thinking and doing, as well.

Building a better experience through data

According to John Ginder, manager for systems analytics with Ford Research & Innovation, the company has been doing advanced business modeling for about 20 years, but big data is something else. Today’s technologies are allowing Ford to handle larger, more-diverse datasets than ever before possible, and its efforts are already beginning to bear fruit in numerous places — including in the cars themselves.

The most obvious example of data influencing the driving experience might be the types of data car companies are actually giving back to drivers. At Ford, its Energi line of plug-in hybrid cars generate 25 gigabytes of data per hour that’s then processed and given back to drivers via a mobile app. It tells them about battery life, the nearest charging stations and other data about the vehicle’s performance.

The MyFord mobile app architecture.

Ginder said all that data is the result of a “convergence of need and opportunity.” The opportunity is a way to experiment with collecting and presenting vehicle data on a group of early adopters that’s probably more interested in this type of advanced technology. The need has to do with what Ginder calls “range anxiety” — when drivers are getting used to electric vehicles, they need reassurance they’re not going to run out juice.

However, Ginder said, the company is just scratching the surface of what’s possible, because there aren’t that many of the electric vehicles on the road yet. The goal is to better understand how drivers are using the vehicles and use that information to continuously improve the vehicles and the overall experience. Ford’s Super Duty line of pickup trucks also offers a “crew chief” package that lets bosses monitor the fuel consumption, engine performance and other data about their fleets of vehicles.

Mike Cavaretta, technical leader for predictive analytics and data mining with Ford Research & Innovation, added that Ford is really interested in collecting more data from more vehicles, but noted there’s also a privacy concern that could come into play. The potential of someone knowing where and how you’re driving might not appeal to the mainstream just yet (just look at all that data Tesla collects about its cars and can present if it really wants to), but as with the Energi, data does present some opportunities to improve the customer experience.

The test cars in Ford’s research labs are collecting about 250 gigabytes of data per hour from high-resolution cameras and an array of sensors, Cavaretta noted, and the company is trying to find out what data is most useful and how it might be rolled into production vehicles.

Building betters cars through data

Of course, sometimes the best data isn’t the stuff you see, but the stuff that just makes your car better. Cavaretta said Ford analyzes a lot of social media and other external data in order to figure out, for example, what customers are saying about their vehicles compared with other makes and what problems they’re having.

Opens with the touch of a foot. Source: Ford

In one recent case, the product development team was curious as to whether the Ford Escape sport-utility vehicle should have a standard liftgate (i.e., it opens manually and the rear window can flip open) or a power liftgate in which the glass and the gate are one piece. In the latter option, the gate opens automatically by tapping under the rear bumper with your foot, but the window doesn’t open at all. Regular surveys hadn’t addressed the question, so Cavaretta and his team took to social media, where people were actually talking about it quite a bit and seemed to heavily favor the power liftgate in most cases. It’s now a feature.

Back in 2004, Ford built a self-learning neural network system for its Aston Martin luxury brand that maintains proper engine function by recognizing engine misfires and particular driving conditions and adjusting warnings and performance accordingly.

Ginder said his team has been improving on that technology ever since and actually expanded its use into a system, called Smart Inventory Management System, that lets dealers ensure they have the optimal stock of vehicles and features on their lots. Historically, he said, some dealers were very sophisticated about inventory management, while others were more reactionary (“They just sold a red Mustang,” he joked, “so they think they need to go order another red Mustang.”) With SIMS, all sorts of data about vehicle sales and other locally relevant data from across the country is aggregated in Ford’s big data platform, and the neural network algorithms learn the current patterns so Ford can make better recommendations — whether or not dealers choose to heed the advice.

Selling big data internally

Cavaretta characterizes the division in which he and Ginder work as “an Ernst & Young, but just for Ford,” an internal consultancy (as opposed to Ford’s more-traditional research and development division) in charge of solving business problems via analytics. About 80 percent of those problems come directly from those lines of business, while about 20 percent are the research division’s own ideas. However, although he’s excited about how big data can help his team answer these questions in novel ways, it’s not always an easy sell with other parts of the company.

Mashing up data sources such as social and sales in order to find insights is a pretty easy sell, Cavaretta explained, but getting people to put sensors in everything and collect data every second or with every transaction can still be a bit challenging. In part, this is just a lingering effect of the constraints that legacy technologies imposed on the company. It wasn’t possible to store all this data, so people just got accustomed to the status quo of summarizing data hourly, for example.

Source: Ford

Now, however, he’s pushing them to “dial it down” and collect data at the lowest level possible and as often as possible. In manufacturing alone, he explained, there are between 20,000 and 25,000 parts in any given vehicle, and there’s a supply chain that spans from parts suppliers all the way up to dealerships. Getting a complete view of this process could help drive serious efficiencies and, Cavaretta said, “We don’t see anything but big data technologies that can get us there.”

Other areas where Ford is collecting, or wants to collect, more real-time data is from websites, call centers and the company’s credit-processing arm, he added.

Building big data internally

In order to accomplish their lofty goals, the Research & Innovation analytics team relies heavily on open source technologies, most prominently Hadoop. However, Cavaretta said, they’ve been experimenting with a variety of natural-language processing tools, too, and even did a proof-of-concept with SAP’s HANA in-memory analytic database. The NLP tools were first turned on text analysis of internal surveys and dealer network documents, but now are used pretty heavily on social media and other web data.

Their team has some systems numbering in the dozens of nodes in its own building, but on weekends it’s able to borrow high-performance computing cycles from Ford’s Numerically Intensive Computing Center next door in order to model recommendation engines and other tasks that demand serious computing power.

But as a part of a specialized research division, the work that Ginder, Cavaretta and their team do on everything from Hadoop to visualization with tools like Tableau isn’t automatically ready for primetime. In fact, Cavaretta said, it looks at “what’s the art of the possible” and tries to show the value of it. It’s like a vanguard, he added, going out and seeing what’s ahead and then reporting back.

At that point, projects are often handed off to Ford’s central IT team that actually puts the technologies into production. A system that took the research team weeks to deploy and start deriving insights from might take IT months to make production-ready. However, Ginder added, his team can’t just throw stuff over the wall and abandon it — it has to collaborate with the IT team and individual departments throughout the project’s lifecycle.

An important part of this cross-company relationship — and something many CIOs have likely heard before — is having data scientists on board that can see the world through the eyes of both technologists and businesspeople, two groups that often have different concerns and goals in mind. “We look for people who can bridge those worlds,” Ginder said. “It’s hard to find these people, but they’re hugely important to organizations.”

Feature image courtesy of Shutterstock user PhotoSmart.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• The importance of putting the U and I in visualization

However, companies are starting to talk openly about what they look for in data scientists, including the skills someone should have and what they’ll need to know to survive an interview. I spent a day at the Predictive Analytics World conference on Monday and heard both Netflix and Orbitz give their two cents. That’s also the same day Hortonworks published a blog post about how to build a data science team.

Granted that “data scientist” is a nebulous term — perhaps as much so as “big data” — these tips (a mashup of all three sources) are still broadly applicable. If you want to make the leap from guy who knows data to data scientist, I suggest paying attention.

1. Know the core competencies.

For most of us, there’s readin, ‘ritin’ and ‘rithmetic. For data scientists, there’s SQL, statistics, predictive modeling and programming (probably Python). If you don’t have at least a grounding in these skills, you’re probably not getting through the door, in part because they form a common language that lets people from different backgrounds talk to each other.

Hortonworks’ Ofer Mendelevitch describes the ideal data scientist as occupying a place on the spectrum between a software engineer and a research scientist. In distinguishing a great engineer, mathematician or data analyst from a data scientist, programming skills are probably the biggest variable. That’s because being able to write code means you’ll have an easier time testing out your hypotheses and algorithms, hacking through certain problems and generally thinking in ways that actually relate to the products your employer is building.

Source: Hortonworks

Chris Pouliot, director of algorithms and analytics at Netflix, said even being able to “pseudo-code” might be good enough if someone is otherwise a strong candidate. You can pick up SQL or Python or whatever you need pretty quickly, he noted.

Or, hinted Orbitz VP of Advanced Analytics Sameer Chopra, you could just suck it up and learn Python now: “If you were to leave today and ask ‘What specific skills should I learn?’: Python.”

2. Know a little more.

Of course, just meeting the minimum requirements never got anybody a job (well, almost nobody). What Pouliot is really looking for in a candidate are: an advanced degree in a quantitative field; hands-on experience hacking data (ideally using Hive, Pig, SQL or Python); good exploratory analysis skills; the ability to work with engineering teams; and the ability to generate and create algorithms and models rather than relying on out-of-the-box ones.

Chopra’s advice was to get up to speed on machine learning, especially if you want to work in Silicon Valley, where machine learning has exploded in popularity. He’s also a big fan of honing those hacking skills because data munging is such a valuable skill when you’re dealing with so many types of data that you need to process so they work together. If you can do quality analytics across myriad data sources, Chopra said, “you can write your own ticket in this day and age.”

Oh, and if you’re planning to work at a startup, he added, R is almost a must-know for anyone whose job will entail statistical analysis.

3. Embrace online learning.

If it all sounds a little daunting, don’t be too worried, Chopra advised. That’s because there are plenty of opportunities to learn these new skills online via both massive open online courses (he’s particularly keen on Udacity’s Computer Science 101 and Andrew Ng’s machine learning course on Coursera) and universities’ own online curricula. Chopra also suggested joining professional groups on LinkedIn, participating in Kaggle competitons and maybe even getting out of the house by going to meetups.

Whatever you’re curious about, though — text mining, natural language processing, deep learning — you can probably find someone willing to teach you for free or nearly free, and any additional skills will help set you apart from the crowd.

4. Learn to tell a story.

Last month at Structure: Data, DJ Patil told me that one of the biggest skill shortcomings in data science is the ability to tell a story with data beyond just pointing to the numbers. Chopra agreed, noting that today’s new visualization tools make it easier to display data in formats that non-scientists might be able to (or at least want to) consume. A corollary of storytelling is good, old-fashioned communication: All the charts in the world won’t make a difference if you can’t communicate to product managers or executives why your findings matter.

Pouliot is a little less sold on communication skills, though — at least sometimes. If you’re an engineer primarily talking to other engineers, he told the room, you probably can speak all the jargon you want. It’s only if someone has a business-facing role when communication really becomes important.

5. Prepare to be tested (aka “Your pedigree means nothing”).

After you’ve learned all these skills, added them to your résumé and talked to a hiring manager about how good you are at them, it’s likely testing time. Prospective Netflix data scientists go through a battery of exercises, Pouliot says, including explaining projects they’ve worked on and questions to determine the depth of their knowledge. They’ll also be asked to devise a framework that solves a problem of the interviewer’s choice.

Chris Pouliot

One thing Pouliot warned about is an over-reliance on what’s on your résumé. Right off the bat, for example, he’ll test the heck out the skills or knowledge that someone claims to ensure they really know it.

Having a Stanford degree and work experience at Google don’t necessarily make someone a shoo-in, either. Pouliot acknowledged during a quick chat after his presentation that he’s been seduced by the perfect resume before — even going so far as to cut a few corners to get someone in for an interview — only to be disappointed in the end. Everyone has to pass the tests, he said, and some of the best applicants on paper crashed and burned very early in the process.

6. Exercise creativity.

It’s during the testing phase at places like Netflix that all those personal skills and experience can come into play. There’s often no right answer when it comes to answering the hypotheticals an interviewer like Pouliot might ask, and he gives bonus points for solutions he’s never seen before. “Creativity is one of the biggest things to look for when hiring data scientists,” he said. Later, he added, “Creativity is king, I think, for a great data scientist.”

Bonus tips for anyone hiring and managing data scientists

Technically, Pouliot’s talk at Predictive Analytics World was about hiring data scientists, but much of the insights were probably more valuable to aspiring data scientists. Some of them, though, we’re definitely for management, possibly at the C-level. A few points to consider:

• Netflix has a standalone data science team that works closely with other departments but ultimately answers to itself. This helps the data scientists collaborate with one another, gives them upward mobility (i.e., they might never become director of marketing, but they could become director of data science) and makes it easier to manage them because everyone speaks the same language so an employee knows his boss knows his stuff.

However, he noted, the alternative approach of embedding data scientists within other departments does bring its own benefits. That type of setup can result in a better alignment of research efforts and business needs, and it can help products get built faster because everyone is on the same page. Pouliot suggests one compromise might be to keep a centralized data science team but locate it physically near the other teams it will be interacting with most often, and other is just to ensure you have representatives from every stakeholder department present for meetings and problem-solving exercises.

• Actually, if you just cannot hire data scientists with all the skills you want them to have, Mendelevitch from Hortonworks suggests a similar tactic. It can be difficult to teach applied math to software engineers and vice versa, so, he writes, “[S]imply build a Hadoop data science team that combines data engineers and applied scientists, working in tandem to build your data products. Back when I was at Yahoo!, that’s exactly the structure we had: applied scientists working together with data engineers to build large-scale computational advertising systems.”

• If you want to retain your good data scientists once you’ve hired them — especially in Silicon Valley where they can walk out the door and get five offers — paying them the market rate is a good start. Additionally, Pouliot said, letting them work on challenging products will keep them happy. Micro-managing them will not.

This post was corrected 4/23 with the correct spelling of Hortonworks’ Ofer Mendelevitch.

Feature image courtesy of Shutterstock user Sergey Nivens.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• 9 Companies that Pushed the Infrastructure Discussion in 2010

A few years into Steve Jobs’s second stint as Apple’s CEO, the company hadn’t yet reinvented itself as one of the world’s most-important technology companies, but it was a big-enough distribution channel for the two-year-old Gracenote. At that point, Gracenote had built a respectable business collecting and providing metadata for the compact discs that people were ripping onto their computers, and it relied on software partners to get in front of the music consumers doing the uploading. One of those partners was a popular Mac jukebox application called SoundJam MP.

Ty Roberts Source: Gracenote

So, Gracenote Co-founder and CTO Ty Roberts told me during a recent interview, his company heeded Apple’s warning and bought more servers. At some point around that time (details on the date of the acquisition are sketchy), Apple bought SoundJam MP. Then, at MacWorld in January 2001, Apple released the first version of iTunes (based on the SoundJam technology) and grew Gracenote’s footprint by putting it on more machines. In October 2001, Apple released the iPod and changed Gracenote’s life forever.

The holiday season — particularly Christmas morning — provides a clear example of how stark the change was. “We used to call it iPod day,” Roberts explained, because the company’s servers would go crazy as people opened up their new iPods and immediately began ripping CDs onto their computers. The company’s chief scientist would stay up 20 hours a day for 5 days straight to make sure the database didn’t crash under the load.

From that point on, Roberts explained, a graph showing the rate at which people were uploading music to Gracenote would go from a steady incline into a vertical line. At one point the company was getting metadata from — by Roberts’s estimate — literally every CD being ripped onto personal computers. There was so much database traffic — both writing and reading — because Apple didn’t release the first version of the iTunes Store until April 2003; if users wanted to use their iPods, they had to upload music first.

Scaling like the big boys

Today, of course, Gracenote (which Sony acquired for $260 million in 2008) is pretty much ubiquitous, at least when it comes to metadata. It has metadata for about 130 million songs — and growing — from all over the world and provides metadata to everything from iTunes to Path to your car’s entertainment console. Even if they’re not available for sale as MP3, if someone somewhere at some point ripped a CD and entered its information, Gracenote has data on those artists and songs.

Its database now gets 15 billion queries a month, or 500 million a day (“We’re probably bigger than Bing,” Roberts joked), and the company’s infrastructure has scaled a few times to meet this demand. What began as a small web database running on a few servers grew into an Oracle environment that provided better performance. And when Oracle became cost-prohibitive because of Gracenote’s expanding scale, it shifted again into a highly optimized system that spans thousands of cores in four global data centers.

Now, GM and VP of Automatic Content Recognition Michael Jeffrey noted, almost everything from the chip level up is optimized specifically for Gracenote.

There’s no “world music” when you’re in the “world”

And this setup lets Gracenote do a lot more than just recognize music listeners’ files and give them the album art. For one, Roberts explained, it lets Gracenote be a global company. “We want to have all the music in the world,” Roberts said, “… because our customers ship their products globally.” In fact, part of the reason it’s now part of Sony is that Sony was distributing Gracenote so widely as part of the music player in its Vaio line of laptops.

In order to ensure that everyone has a natural experience wherever they’re accessing Gracenote, part of the job of the company’s 100-person editorial team is to categorize music hierarchically by locality. So, when a user in Japan uploads a CD and Gracenote returns the metadata, it’s categorized as “rock and roll,” for example, rather than a catch-all category like “world music” that a U.S. user might see.

“We want music to feel like a person in your country actually organized it,” Roberts said, “not some dude from California.”

Better music and television through data science

All that data also makes Gracenote a natural fit for recommending new music, although right now the company prefers to let partners handle the algorithms because recommendations tend to be highly product-specific. For example, the iTunes Genius feature is a pretty run-of-the-mill recommendation engine, but, Roberts explained, Apple places a premium on accuracy because its recommendations cost users 99 cents (or more) a shot. With a subscription service like Spotify, though, trying new music is risk-free, so it can play a little faster and looser with its algorithms.

Because Gracenote is present in so many cars — about 35 million — the company has put a lot thought into how to optimally deliver services there, too. Until drivers can bring their interest graphs and music libraries with them to their cars, he explained, any sort of in-car recommendation engine has to be pretty simple and non-distracting — perhaps like thumbs-up or thumbs-down button on the display that will eventually be able to recognize someone’s tastes.

The company has even developed what Roberts calls “machine listening,” which is the ability of an algorithm to recognize the mood, tempo and other audio attributes of music. This is comparable to what Pandora offers, but Gracenote has data on pretty much any song someone could possibly have, which means it can make even your personal music library that much smarter. One idea the company is tinkering is something Roberts describes as “audio coffee.” Depending on any variety of factors — time of day, location, driving conditions or behavior — the stereo system could pick music that either picks up a driver’s pulse or maybe relaxes him.

For Gracenote’s next chapter, the company is banking tablets to deliver a kick like the iPod did last decade. Gracenote is already working with television partners on real-time ad-swapping and intelligent content recommendations, and now it wants to dive deep into the second-screen world. Its new product called Entourage uses a tablet’s internal sensors to hear the television show or music playing in a room and then surface related content, perhaps from the web — like what Entourage user Zeebox provides — or perhaps produced, interactive material like the SyFy channel delivers via its Sync app.

Two ads for two different viewers.

Later this year, GM and VP Jeffrey said, Gracenote will be doing pilots with some large sports broadcasters around a “cheer and jeer” feature that measures how hard people in a room are cheering for or booing their favorite sports teams. If you’re elated, you might see an ad for season tickets. If you’re sad, maybe it’s an an for beer.

Even Roberts is impressed, especially considering that the company’s first use of audio recognition was to make sure users got the right data for their exact version of a song: “I never thought the recognition would break open these kind of new fields.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Forecast: the future of the digital music industry
• Social-TV apps and consumer behavior

These individuals and companies aren’t entirely bad, of course, and they’re actually necessary. Apps that help predict what we want to read, where we’ll want to go next or what songs we’ll like are certainly cool and even beneficial in their ability to automate and optimize certain aspects of our lives and jobs. In the corporate world, there will always be data experts who are smarter and trained in advanced techniques and who should be called upon to answer the toughest questions or tackle the thorniest problems.

Last week, for example, Salesforce.com introduced a new feature of its Chatter intra-company social network that categorizes a variety of data sources so employees can easily find the people, documents and other information relevant to topics they’re interested in. As with similarly devised services — LinkedIn’s People You May Know, the gravitational search movement, or any type of service using an interest graph — the new feature’s beauty and utility lie in its abstraction of the underlying semantic algorithms and data processing.

The problem, however, comes when we’re forced to rely on these people, features and applications to decide how data can affect our lives or jobs, or what questions we can answer using the troves of data now available to us. In a true data democracy, citizens must be empowered to make use of their own data as they see fit and they must only have to rely apps and experts by choice or when the task really requires an expert hand. At any rate, citizens must be informed enough to have a meaningful voice in bigger decisions about data.

The democratic revolution is underway

The good news is that there’s a whole new breed of startups trying to empower the data citizenry, whatever their role. Companies such as 0xdata, Precog and BigML are trying to make data science more accessible to everyday business users. There are next-generation business intelligence startups such as SiSense, Platfora and ClearStory rethinking how business analytics are done in an area of HTML5 and big data. And then there are companies such as Statwing, Infogram and Datahero (which will be in beta mode soon, by the way) trying to bring data analysis to the unwashed non-data-savvy masses.

Combined with a growing number of publicly available data sets and data marketplaces, and more ways of collecting every possible kind of data —  personal fitness, web analytics, energy consumption, you name it — these self-service tools can provide an invaluable service. In January, I highlighted how a number of them can work by using my own dietary and activity data, as well as publicly available gun-ownership data and even web-page text. But as I explained then, they’re still not always easy for laypeople to use, much less perfect.

Statwing spells out statistics for laypeople.

Can Tableau be data’s George Washington?

This is why I’m so excited about Tableau’s forthcoming IPO. There are few companies that helped spur the democratization of data over the past few years more than Tableau. It has become the face of the next-generation business intelligence software thanks to its ease of use and focus on appealing visualization, and its free public software has found avid users even among relative data novices like myself. Tableau’s success and vision no doubt inspired a number of the companies I’ve already referenced.

Assuming it begins its publicly traded life flush with capital, Tableau will not just be financially sound — it will also be in a position to help the burgeoning data democracy evolve into something that can last. More money means being able to develop more features that Tableau can use to bolster sales (and further empower business users with data analysis), which should mean the company can afford to also continually improve its free service and perhaps put premium versions in the hands of more types of more non-corporate professionals for free.

Tableau is already easy (I made this) — but not easy enough.

The bottom-up approach has already proven very effective in the worlds of cloud computing, software as a service and open-source software, and I have to assume it’s a win-win situation in analytics, too. Today’s free users will be tomorrow’s paying users once they get skilled enough to want to move onto bigger data sets and better features. But the base products have to be easy enough and useful enough to get started with, or companies will only have a lot of registrations and downloads but very few avid users.

And if Tableau steps ups its game around data democratization, I have to assume it will up the ante for the company’s fellow large analytics vendors and even startups. A race to empower the lower classes on the data ladder would certainly be in stark contrast to the historical strategy of building ever-bigger, ever-more-advanced products targeting only the already-powerful data elite. That’s the kind of revolution I think we all can get behind.

Feature image courtesy of Shutterstock user Tiago Jorge da Silva Estima.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• 4 iPad apps to help wrangle data