Earlier this year Lockheed Martin shared details of its implementation of a D-Wave Systems quantum computer, which reportedly cost $10 million: The contractor is using the computer to develop new aircraft, radar and space systems.

Now Google is taking steps at incorporating more quantum computing into its operations with the Quantum Artificial Intelligence Lab, which will be located at the NASA Ames Research Center in Moffett Field, Calif. Researchers from the Universities Space Research Association will be able to use the machine 20 percent of the time, Forbes reports. That could lead to lots of interdisciplinary thinking and collaboration.

For Google, though, the goal of the initiative is to make strides in machine learning, according to a Thursday Google Research blog post. The best results could trickle down to end users, perhaps in search results and speech-recognition applications.

Quantum computing could mean smarter smartphones

Google has already assembled machine-learning algorithms that involve quantum elements, Hartmut Neven, a Google director of engineering, explained in the post:

One produces very compact, efficient recognizers — very useful when you’re short on power, as on a mobile device. Another can handle highly polluted training data, where a high percentage of the examples are mislabeled, as they often are in the real world.

It’s not hard to imagine how quantum computing could inform machine learning on a smartphone with just a drop of battery life left. It could be that a smarter smartphone one day will take a minuscule amount of input and determine with a high probability who a user wants to talk to or what information it needs right away, rather than forcing the user to cycle through a string of commands and risking the death of the battery altogether.

The applications might have arisen after Google’s earlier partnership with D-Wave, which came to light in a different blog post from Neven in 2009.

Google has already used machine learning to recognize faces and other things in photos and videos. New technology Google executives talked about at the Google I/O developer conference in San Francisco on Wednesday also appears to use machine learning to stitch together photos and clean them up.

What Google has learned so far is the best results come from blending regular binary computing using ones and zeros with quantum style computing. Quantum computing accommodates the space between a one and a zero with quantum bits of information, or qubits. It can express likelihood as well as take shortcuts by approximating when handling certain kinds of workloads. Given what Google has observed thus far, it could decide to build hardware combining quantum and classical computing capabilities.

For now, though, Google is diving deeper into quantum computing with the D-Wave machine. The move could kick off a sort of arms race for webscale companies to buy quantum computers and come up with new notions by way of probabilistic logic. In this way, Google could help push the development of quantum computing much like its invention of MapReduce changed the way firms do distributed data processing.

In any case, quantum computing has a long way to go before reaching commercial viability. That could take decades (so far it has). But because the organization at the helm of the quantum research is Google and not IBM or Bell Labs, regular people could start seeing much more of the advantages in just a few years’ time, which in turn could drive commercialization.

Feature image courtesy of Shutterstock user pixeldreams.eu.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Takeaways from the second quarter in cloud and data
• GigaOM Euro 20: the European startups to watch

There are two primary reasons why, the first of which is obvious: Companies are tired of managing applications and infrastructure, so something that optimizes a common task using techniques they don’t know on servers they don’t have to manage is probably compelling. It’s called cloud computing.

The other reason is that the big part of big data really is important if you want to get a really clear picture of what’s happening in any given space. While no single end-user company can (or likely would) address search-engine optimization, for example, by building a massive store comprised of data from hundreds or thousands of companies as well as the entire web, a cloud service dedicated to that specific task can.

From web security to systems management, we’re already seeing how centralized data stores provide SaaS companies a broad view into what’s happening that can then be filtered down to serve each individual customer’s specific situation. BloomReach, a SaaS startup that helps companies optimize web-page content, is another good example of this principle in action.

How do you say, “cotton maxi dress”

Ideally, BloomReach Head of Marketing Joelle Kaufman told me, the company wants to help customers ensure they get found in web searches by making sure they’re not invisible (buried deep down), irrelevant (not saying anything meaningful on their sites) or incompatible (not speaking their consumers’ language). On Tuesday, the company announced a new feature called Continuous Quality Management, which lets customers continuously monitor their pages to ensure they’re still featuring the right products and the right terminology. It’s the latest addition to a seemingly useful service that’s built atop a big data foundation few — if any — of its customers would ever attempt to build themselves.

BloomReach is able to help companies optimize their sites because it’s constantly crawling the web in order to figure out how everyone else is describing their content, laying out their pages and structuring their links. Running on the Amazon Web Services cloud, BloomReach runs more than 1,000 Hadoop jobs a day that process about 5 terabytes of data and a billion data points about users’ site behavior. With the latter, co-founder and CTO Ashutosh Garg explained, the company is trying to figure out who’s visiting sites, what they’re doing, how long they’re spending there and how they’re related in terms of behavior.

“You need to have the right amount of data and from the right places before we can do anything with it,” he said. “… It’s a massive machine learning problem.”

When you consider all the possible ways something could be described or formatted, the scale of the problem becomes more evident. Simple semantic analysis like associating “desk” and “table” is easy, Garg explained, but what if some wants a lightweight camera and you only have its exact weight listed without any indication of how it compares to other options? What if people searching for “smartphones” really mean “Android phones,” but you’re top-loading your results with BlackBerry phones and Windows phones?

Another of Garg’s hypotheticals has to do with consumers’ presentation biases. If, for example, they’re looking at a lot of websites that look the same or focus on the same things (e.g., megapixels for digital cameras), they’ll expect to see the same things from every site.

10 nonillion possibilities: Choose 1.

From a sheer numbers perspective, things get even hairier when you’re trying to determine the relationship between any two pages in order to figure out the best path for links to to take. Garg said this is what computer scientists call an NP-complete problem, which means the amount of time it takes to process the results is exponentially greater than the amount of content you’re analyzing. So, for example, analyzing 40 pages doesn’t take 10 times as long as analyzing 4 pages, but more like 100 times longer.

Actually, BloomReach CEO Raj De Datta gave me another example of this problem when we spoke in early 2012. Here’s how I described it then:

[I]f a company wants to display just 1,000 products across 100 pages, De Datta explained, there are 10-to-the-28th-power (10 octillion) possibilities for how to do that. When it comes time to describe those products, there are 10-to-the-30th-power (10 nonillion) possibilities.

If a website has a million pages, Garg said, “it will take you longer than the life of the universe to solve that problem.”

Where this type of problem arises, BloomReach turns to Monte Carlo simluations, a favorite technique of physicists and Wall Street quants. The method involves running lots of simulations over large data sets in order to determine approximate results in a reasonable time frame. (And if all this isn’t enough computer science and cloud infrastructure for you, I suggest attending our Structure conference in June, which features a who’s who list of speakers, including Google’s Jeff Dean, Facebook’s Jay Parikh and Netflix’s Adrian Cockroft.)

Different queries, different pages

Things get even trickier when you’re trying to change the content of web pages in real time as people are searching for things. This isn’t the best method for organic search, where pages need to stay pretty consistent with the indexed versions, but it can be ideal in situations such as paid search and mobile. There are millions of ways to segment buyers, Garg explained, and how accurately you assess their intent and display your content can make the all the difference. Whether someone is a new or repeat visitor often matters, as does whether someone is price-conscious (e.g., the query included “cheap”) or perhaps searching for a particular brand.

Source: BloomReach

Around the holidays, the company actually realized something interesting: The bounce rate on queries for things like “gifts for dad” or “gifts for co-workers” was pretty high, but so was the conversion rate. The time to conversion was relatively fast, as well. It turns out, Garg explained, that people don’t like to overthink certain gifts too much, so if something is presented in a visually appealing manner and is within their price range, they’ll buy.

But creating these types of models involves more than meets the eye. For all the talk about machine learning — and machines do a majority of the work for BloomReach — people also play a critical role. A person might know better than a machine whether something was likely purchased as gift, Garg explained, or they might spot the offensive content on the T-shirt the machine decided was ideal.

“Humans are really good at creativity, thinking through stuff,” he said.

Smart humans are also good at knowing when they’re overmatched, which is why SaaS is so valuable in the big data era. CMOs could try doing what BloomReach or similar companies such as DataPop are doing, or they could pay someone to do it much better. Guess which route the smart ones will take.

Feature image courtesy of Shutterstock user Andrea Danti.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

When you’re in Atlanta, as I was during a recent trip, the obvious place to start talking about orange juice and data is with Coca-Cola. Foreman can tell you all about how the beverage giant — whose headquarters tower over the city just a just a mile away from MailChimp’s office — uses advanced algorithms and giant vats of different juices to ensure the proper flavor of its Simply Orange line of orange juice. However, it’s something else Coca-Cola is doing that inspired the way Foreman thinks about data and that’s helping MailChimp re-imagine what it means to engage with fans, readers and customer through their inboxes.

Anyone familiar with how large web companies came to pioneer the practice of what we now call “big data” should appreciate the analogy. Coca-Cola, which also owns Minute Maid, produces a lot of excess pulp when it makes orange juice. For decades, presumably, it had just been throwing that pulp away, but in 2006 it decided to make use of it by launching a new product called Minute Maid Pulpy. Sold primarily in Asian countries, Pulpy has become a billion-dollar business for Coca-Cola.

Once MailChimp is done with its primary business of sending emails, it has a lot of pulp of its own in the form of data. And rather than just ignoring it or writing up some cute blog posts (which he also does), Foreman and his bosses want to turn that data into revenue.

First things first: Making better orange juice

Neil Bainton

Actually, though, MailChimp first brought in Foreman in 2011 to help the company improve its core business of letting users build and send their emails. MailChimp’s culture was built around many things, COO Neil Bainton told me, but data wasn’t one of them. It had “various fits and starts” through the years trying to work data into its business model, and each step just added more complexity.

The challenges were technological as well as cultural, but Foreman had a plan, of which focus was a key aspect. Keeping a tight focus meant Foreman and his lone-developer sidekick could build what they needed to in a short timeframe. It also meant the company didn’t have to worry about some massive overnight transformation into a data-obsessed company like Google.

John Foreman

“[They] don’t need to be afraid the entire culture is gonna fall down if we bring in this weird math guy,” he joked.

Foreman’s first project — deploying artificial intelligence models that would automatically detect spammy email lists from MailChimp’s users – is actually critical to the way MailChimp operates, though. It was up and running in production within a year, after a technologically challenging effort of merging separate database instances for each customer into a single environment that would let MailChimp run complex analyses across its customer base.

It’s such an important project, Foreman explained, because internet service and email providers keep reputation scores on the IP addresses that send email through their systems. Because MailChimp serves as the email engine for its millions of users, sending too many messages that get flagged as spam and lower MailChimp’s reputation will have a negative impact on everyone. The company used to deal with spam manually, and only after recipients began complaining about the messages they received.

“It used to be before we had that AI model in place that everyone had a crappier experience,” Foreman said.

Say goodbye to those ’90s fans, Pearl Jam

Source: MailChimp

Now, however, MailChimp knows some of the telltale signs of spam for which it should be on the lookout. If too high a percentage of email addresses on a given list are also available via publicly available lists or those you can buy on sketchy corners of the internet, it’s probably spam. Too many old and far-more-likely-to-be-dead Earthlink or Compuserve addresses, or letters within one keystroke of each other as if someone just mashed the keyboard? Probably spam.

Thankfully, though, about 98 percent of the spam that MailChimp identifies is what Foreman calls “ignorant” — that is, people or companies that just don’t know the laws or best practices around sending emails. But ignorance doesn’t mean MailChimp relaxes its rules. Recently, it even flagged Pearl Jam for spammy practices because the band was trying to reconnect with old fans whose email addresses read like a who’s who list of 1990s email providers.

Having such a high percentage of ignorant spam actually has a positive effect on the company’s overall goal of monetizing its vast data repositories. Because the AI model automates what used to be a manual process, and because most innocent spammers will fall in line quickly once they’re notified (as opposed to nefarious spammers who constantly try to outsmart the system), MailChimp can pretty much set the model loose, forget about it and get to work on new efforts, Foreman said.

Now, about that pulp

Spam under control, MailChimp can focus its efforts on actually building new products with data, just like Coca-Cola did with that extra pulp. One of its first orders of business is figuring out how to help customers get to know better the people to whom they’re sending their newsletters.

With this in mind, the company built a service called Wavelength that shows customers other newsletters that are similar to theirs. But the system that powers Wavelength also stores pretty much every interaction that every email address in the company’s database has with the newsletters they’re sent. That means what emails they open and when they open them, what links they click and when they click them, and what other newsletters they’re subscribed to. MailChimp also has a feature called Ecommerce360 that lets customers track clicks right through to conversions (marketing speak for someone actually buying something).

The company has been playing around with this data to identify clusters of users based on their behaviors and their interests — some of which Foreman has detailed on the company’s blog — and now it wants to roll it out to customers via a product MailChimp is calling ChimpQuery. Built atop Google’s BigQuery analytics service, ChimpQuery will let customers start doing this type of clustering and segmentation on their own, while saving MailChimp the troubles of hosting that infrastructure itself. (You can play with a monstrous, interactive graph of the entire MailChimp subscriber list here.)

If you sell knitting supplies and you find out there’s a big cluster of people on your mailing list who also are interested in wedding planning and custom jewelry, there might be an opportunity to create your content with these interests in mind or even to partner with companies in those spaces.

A sample cluster of subscribers.

Another topic that has been on Foreman’s mind lately is what he calls “frequency elasticity of engagement.” He’s done research suggesting that blasting the heck out of your email list might actually have detrimental effects in the long term (regardless of how the Obama campaign successfully exploited this strategy) but noted that engagement also has a lot to do with content and a particular company’s given user list. MailChimp’s data could help customers figure out the ideal schedule for emailing their subscribers.

For example, Birchbox has really high engagement because people love the service and have to open their emails to find out what goodies they’re receiving. Emails from a company like Papa John’s, on the other hand, might sit in someone’s inbox essentially as spam until they want to order a pizza and go searching for a coupon. Everyone has to figure out what pace and engagement metrics work for them.

Reining expectations back in

However, now that management is fully sold on the power of data, Foreman sometimes finds himself managing expectations rather than just pitching his ideas. COO Bainton, for example, is adamant that MailChimp start aiding its publishing-industry customers by using techniques such as natural-language processing and semantic analysis to help them personalize emails based on readers stated and unstated interests (that is, what boxes they check when they sign up and what stuff they actually click on).

Foreman, well, he’s pretty sure that’s too big a challenge for MailChimp to tackle considering how many publishing customers it has. MailChimp would have to understand all those customers’ industries to some degree (open source tools tend to highlight technically but not situationally relevant relationships, he said, and don’t always understand things like sarcasm) and probably the different languages they publish in, as well. Rather than understand content, he’d rather focus personalization efforts around how users are connected.

The company also needs to balance its ambitions with what’s legally and socially acceptable. The creep factor might be more important than what’s legal when it comes to email marketing. MailChimp determines the legality of everything it does before rolling it out, Foreman explained, but in era of “post-modern spam” where legitimacy is in the eye of the recipient and where some people use their “spam” button as a proxy for unsubscribing, companies must be careful not to offend.

“The more we can tell you about that list without getting creepy is really useful,” Bainton said. However, he added, ”I think expectation is more important than law.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the next front in big data might be psychological
• Will Hadoop Vendors Profit from Banks’ Big Data Woes?
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

Machine learning is such a hot topic right now because data volumes are becoming so large and complex that humans alone can’t query their ways through them fast enough or intelligently enough to spot latent patterns among the mess of data. It’s the algorithmic engine that powers a bunch of Google services and your Netflix recommendations, as well as web content-curation service Prismatic and alternative-underwriting platform ZestFinance. As we covered in some detail at this year’s Structure: Data conference, machine learning is particularly powerful when its ability to correlate tens of thousands of variables is paired with human judgment about what really matters.

Skytree co-founder Alexander Gray (second from left) at Structure: Data 2012. (c) Pinar Ozger

Skytree, for its part, sells a product called Skytree Server that lets users run a wide variety of machine learning algorithms across whatever data they have. It might be an oversimplification, but Skytree is essentially a souped-up version of statistical-analysis packages like SPSS or SAS that’s designed to run fast — and, more importantly — without sampling across a scale-out server architecture. In March, the company also rolled out the beta version of a new product called Adviser that can run on a laptop and walks more-novice users through the analysis of their data, including what methods were used and why, and whether the findings are statistically significant.

I suspect we’re just seeing the opening salvo in what will be a rush to fund machine learning startups over the next couple of years. Skytree is among a number of increasingly promising startups in the space, including (but certainly not limited to) Ayasdi and Quid. As more individuals see the promise of machine learning and get skilled in applying it to their particular problems and datasets — as UPS apparently has — it could become become one of the go-to analytic methods in the big data era.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Health care and big data in 2012
• Takeaways from the second quarter in cloud and data

With that line, the 2009 paper “The Unreasonable Effectiveness of Data” (co-authored by Google co-workers Alon Halevy and Fernando Pereira), Google Director of Research Peter Norvig all but guaranteed his status as one of most-quoted — or at least most-paraphrased — people in the world of big data. Last week, Norvig — as well as Google VP of Energy Arun Majumdar — was bestowed a slightly more-formal honor, as he was inducted into the American Academy of Arts and Sciences.

Norvig, who previously led Google’s search algorithms team and was head of computational sciences at the NASA Ames Research Center, is best known for his work in the realm of artificial intelligence. In fact, the above quote and the paper in which it appears are essentially a testament to the advances Google has been able to make in AI and machine learning thanks to the massive web page and search dataset that Google has amassed. The more examples it has of words and phrases used together in natural language, the better it can perform semantic analysis to determine what’s related to what.

Norvig and Majumdar are among 198 new inductees into the American Academy of Arts and Sciences’ latest class. According to a blog post from Google, they also join six other Google employees as members: Sergey Brin, Larry Page, Eric Schmidt, Vint Cerf, Alfred Spector, Hal Varian and Ray Kurzweil.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Listening platforms: finding the value in social media data
• Takeaways from the second quarter in cloud and data

Redpoint Ventures led the Series A round, and Andreessen Horowitz, the Social+Capital Partnership, Sutter Hill Ventures and others also contributed.

Based in Palo Alto, Calif., the company focuses on sales operations that keep their leads in Salesforce and other cloud-based tools such as Eloqua and Marketo, although Infer Co-founder and CEO Vik Singh said it’s also possible for the software to hook in to on-premise appliances.

Once signed up with Infer — Box, Jive Software, Tableau Software, Yammer, Zendesk and other companies are already paying customers — the system inspects historical sales information to check which deals have been sealed and which fell apart. That becomes training data for a model that scans “hundreds of signals of external data” to get a sense of which potential leads the company stands a chance of closing. Inputs include news articles, social-media accounts, website-traffic data, industry data, financial data, legal data, trademark data — “anything we can get that can give us more of a complete picture on who the customer is,” Singh said. Users can determine the weight of certain types of information.

Users also set priorities for the scoring of leads. “Do you want a model where the higher the score, the more likely (you are) to win (a deal), or do you care more about conversion, or do you care more about lifetime revenue, or deal size? We build the model based on that,” Singh said. It’s not just a neat way to prioritize leads; Singh said many Infer customers have boosted conversion rates with the tool.

Infer data integrated in Salesforce.com

Before starting Infer, Singh spent some time at Google, where he focused on machine-learning methods for automatically providing answers to questions users type into the search box. Another former Googler, one-time chief information officer Douglas Merrill, co-founded a different company that uses lots of external data to make determinations: ZestFinance, formerly known as ZestCash, extracts information from 70,000 sources to figure out if a lender should make a loan.

The broader strategy the two companies have in common — making predictions based on data — has become more popular in recent years, as companies merge and grow data sets to create more than the sum of their parts. In this case, it seems that the approach could garner wide adoption as a few companies optimize the time of their sales and marketing employees and pull ahead of their competitors, and other companies might want to do the same thing to catch up.

Of course, if Salesforce or Oracle acquires Infer, then adoption could come even faster.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The 2013 task management tools market
• Social 2013: The enterprise strikes back
• Startup growth and the new recruiting ecosystem

However, companies are starting to talk openly about what they look for in data scientists, including the skills someone should have and what they’ll need to know to survive an interview. I spent a day at the Predictive Analytics World conference on Monday and heard both Netflix and Orbitz give their two cents. That’s also the same day Hortonworks published a blog post about how to build a data science team.

Granted that “data scientist” is a nebulous term — perhaps as much so as “big data” — these tips (a mashup of all three sources) are still broadly applicable. If you want to make the leap from guy who knows data to data scientist, I suggest paying attention.

1. Know the core competencies.

For most of us, there’s readin, ‘ritin’ and ‘rithmetic. For data scientists, there’s SQL, statistics, predictive modeling and programming (probably Python). If you don’t have at least a grounding in these skills, you’re probably not getting through the door, in part because they form a common language that lets people from different backgrounds talk to each other.

Hortonworks’ Ofer Mendelevitch describes the ideal data scientist as occupying a place on the spectrum between a software engineer and a research scientist. In distinguishing a great engineer, mathematician or data analyst from a data scientist, programming skills are probably the biggest variable. That’s because being able to write code means you’ll have an easier time testing out your hypotheses and algorithms, hacking through certain problems and generally thinking in ways that actually relate to the products your employer is building.

Source: Hortonworks

Chris Pouliot, director of algorithms and analytics at Netflix, said even being able to “pseudo-code” might be good enough if someone is otherwise a strong candidate. You can pick up SQL or Python or whatever you need pretty quickly, he noted.

Or, hinted Orbitz VP of Advanced Analytics Sameer Chopra, you could just suck it up and learn Python now: “If you were to leave today and ask ‘What specific skills should I learn?’: Python.”

2. Know a little more.

Of course, just meeting the minimum requirements never got anybody a job (well, almost nobody). What Pouliot is really looking for in a candidate are: an advanced degree in a quantitative field; hands-on experience hacking data (ideally using Hive, Pig, SQL or Python); good exploratory analysis skills; the ability to work with engineering teams; and the ability to generate and create algorithms and models rather than relying on out-of-the-box ones.

Chopra’s advice was to get up to speed on machine learning, especially if you want to work in Silicon Valley, where machine learning has exploded in popularity. He’s also a big fan of honing those hacking skills because data munging is such a valuable skill when you’re dealing with so many types of data that you need to process so they work together. If you can do quality analytics across myriad data sources, Chopra said, “you can write your own ticket in this day and age.”

Oh, and if you’re planning to work at a startup, he added, R is almost a must-know for anyone whose job will entail statistical analysis.

3. Embrace online learning.

If it all sounds a little daunting, don’t be too worried, Chopra advised. That’s because there are plenty of opportunities to learn these new skills online via both massive open online courses (he’s particularly keen on Udacity’s Computer Science 101 and Andrew Ng’s machine learning course on Coursera) and universities’ own online curricula. Chopra also suggested joining professional groups on LinkedIn, participating in Kaggle competitons and maybe even getting out of the house by going to meetups.

Whatever you’re curious about, though — text mining, natural language processing, deep learning — you can probably find someone willing to teach you for free or nearly free, and any additional skills will help set you apart from the crowd.

4. Learn to tell a story.

Last month at Structure: Data, DJ Patil told me that one of the biggest skill shortcomings in data science is the ability to tell a story with data beyond just pointing to the numbers. Chopra agreed, noting that today’s new visualization tools make it easier to display data in formats that non-scientists might be able to (or at least want to) consume. A corollary of storytelling is good, old-fashioned communication: All the charts in the world won’t make a difference if you can’t communicate to product managers or executives why your findings matter.

Pouliot is a little less sold on communication skills, though — at least sometimes. If you’re an engineer primarily talking to other engineers, he told the room, you probably can speak all the jargon you want. It’s only if someone has a business-facing role when communication really becomes important.

5. Prepare to be tested (aka “Your pedigree means nothing”).

After you’ve learned all these skills, added them to your résumé and talked to a hiring manager about how good you are at them, it’s likely testing time. Prospective Netflix data scientists go through a battery of exercises, Pouliot says, including explaining projects they’ve worked on and questions to determine the depth of their knowledge. They’ll also be asked to devise a framework that solves a problem of the interviewer’s choice.

Chris Pouliot

One thing Pouliot warned about is an over-reliance on what’s on your résumé. Right off the bat, for example, he’ll test the heck out the skills or knowledge that someone claims to ensure they really know it.

Having a Stanford degree and work experience at Google don’t necessarily make someone a shoo-in, either. Pouliot acknowledged during a quick chat after his presentation that he’s been seduced by the perfect resume before — even going so far as to cut a few corners to get someone in for an interview — only to be disappointed in the end. Everyone has to pass the tests, he said, and some of the best applicants on paper crashed and burned very early in the process.

6. Exercise creativity.

It’s during the testing phase at places like Netflix that all those personal skills and experience can come into play. There’s often no right answer when it comes to answering the hypotheticals an interviewer like Pouliot might ask, and he gives bonus points for solutions he’s never seen before. “Creativity is one of the biggest things to look for when hiring data scientists,” he said. Later, he added, “Creativity is king, I think, for a great data scientist.”

Bonus tips for anyone hiring and managing data scientists

Technically, Pouliot’s talk at Predictive Analytics World was about hiring data scientists, but much of the insights were probably more valuable to aspiring data scientists. Some of them, though, we’re definitely for management, possibly at the C-level. A few points to consider:

• Netflix has a standalone data science team that works closely with other departments but ultimately answers to itself. This helps the data scientists collaborate with one another, gives them upward mobility (i.e., they might never become director of marketing, but they could become director of data science) and makes it easier to manage them because everyone speaks the same language so an employee knows his boss knows his stuff.

However, he noted, the alternative approach of embedding data scientists within other departments does bring its own benefits. That type of setup can result in a better alignment of research efforts and business needs, and it can help products get built faster because everyone is on the same page. Pouliot suggests one compromise might be to keep a centralized data science team but locate it physically near the other teams it will be interacting with most often, and other is just to ensure you have representatives from every stakeholder department present for meetings and problem-solving exercises.

• Actually, if you just cannot hire data scientists with all the skills you want them to have, Mendelevitch from Hortonworks suggests a similar tactic. It can be difficult to teach applied math to software engineers and vice versa, so, he writes, “[S]imply build a Hadoop data science team that combines data engineers and applied scientists, working in tandem to build your data products. Back when I was at Yahoo!, that’s exactly the structure we had: applied scientists working together with data engineers to build large-scale computational advertising systems.”

• If you want to retain your good data scientists once you’ve hired them — especially in Silicon Valley where they can walk out the door and get five offers — paying them the market rate is a good start. Additionally, Pouliot said, letting them work on challenging products will keep them happy. Micro-managing them will not.

This post was corrected 4/23 with the correct spelling of Hortonworks’ Ofer Mendelevitch.

Feature image courtesy of Shutterstock user Sergey Nivens.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• 9 Companies that Pushed the Infrastructure Discussion in 2010

If you’re like me, though, all you see are the bright lights ahead. They might be some sort of data nirvana, or they might be a privacy-destroying 18-wheeler bearing down on us. They might be both. But we’re going to find out, and we’re we’re going to find out sooner rather than later.

This is because there are small pockets of technologists who are letting their imaginations lead the way. In a suddenly cliché way of saying it, they’re aiming for 10x improvement rather than 10 percent improvement. They can do that because they now have a base set of analytic technologies and techniques that are well positioned to solve, with relatively little effort, whatever data problems are thrown their way.

Here are some themes from our just-concluded Structure: Data conference that I think highlight the promise of data, but also the challenges that lie ahead.

Man and machine unite

Machine learning is already infiltrating nearly every aspect of our digital lives, but its ultimate promise will only be realized when it becomes more human. That doesn’t necessarily mean making machines think like human brains (although, granted, that’s a vision currently driving billions of research dollars), but just letting people better interact with the systems and models trying to discover the hidden patterns in everything around us.

Whatever shape it takes, the results will be revolutionary. We’ll treat diseases once thought untreatable, tackle difficult socio-economic and cultural issues, and learn to experience the world around in entirely new ways. Maybe that consumer-experience scourge known as advertising might actually become helpful rather than annoying.

That would really be something.

Data science, or data intelligence?

I’m not sure there needs to be a distinction between data science and data intelligence, but the latter does connote a grander goal. It’s about trying to solve meaningful problems rather than just serving ads; about trying to understand why things happen just as well as when they’ll happen. This means learning to work with smaller, messier data than we might like — certainly smaller and messier than the data sets underneath most of the massive web-company data science undertakings.

But just think about being able to go beyond predictive models and into a world of preventative — or even professorial — models. If you know what I like, where I go and who my friends are, it might be fairly easy to predict what I want to buy. Figuring out how my decision to buy something might affect my overall well-being and then telling me why? That’s a little more difficult and a lot more beneficial.

Telling stories with data

Have you ever looked at a chart and wondered what the heck it was supposed to be telling you? Or downloaded a report of your Facebook activity only to ask yourself if all the disparate data points come together to paint a bigger picture? Or tried — and failed — to stop a terrorist before his movement to recruit an army of followers gained critical mass?

A big problem with a lot data analysis right now is that it still treats data points as entities unto themselves, largely disconnected from those around them. However, data needs context in order to be really useful; it’s context that turns disparate data points into a story. Don’t just tell me how many steps I took today or the time of day I’m most active on Facebook, but tell me how that relates to the rest of my life.

And don’t just tell me that someone said he wants to kill Americans. Rather, tell me a story about how much more frequently he’s saying it and how much more inciteful his words are becoming.

The internet of things knows all

The mobile phone in your pocket is tracking your every movement and can also monitor the sounds that are surrounding you. That fitness tracker you’re wearing is identifying you by how you walk. Your smart meter data shows when you’re home, when you’re away and when you’re in the shower. Sensors in everything from toothbrushes to cars are quantifying every aspect of our lives.

This volume of data can still be a lot to deal with in terms of its volume, velocity and variety, and we’re still not quite sure what to do with it even if the right tools were in place. But all sorts of entrepreneurs, powerful institutions and intelligence agents have ideas. The technological pieces are coming along nicely, too. Just sayin’ …

This semantic life

The semantic web lives on; only it’s spreading well beyond our search engines and even our web browsers. Soon enough, we’ll be able to surface relevant content and people simply by highlighting passage of text in whatever we’re reading — web page or not — on any type of device. When we speak to our devices, they’ll not only know what we’re saying, but also what we really want even without the help of specific commands or keywords.

That’s a powerful proposition in a world where we increasingly expect our interactions to be hands-free and our answers to come as fast as our questions. Of course, what’s powerful in the hands of consumers driving in their cars or sitting on their couches is even more powerful in the hands of doctors trying to diagnose difficult diseases or aid workers trying lend a helping hand in places where they don’t know the customs or even speak the language.

Feature image courtesy of Shutterstock user GrandeDuc.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout

Human intervention is critical at multiple layers, from choosing the algorithms to apply to feature creation to crafting the entire structure within which a machine will learn, said Scott Brave, founder and CTO of Baynote, at GigaOM’s Structure: Data conference Wednesday.

Down the road, he said, there will be more opportunities for machine-man collaboration, as data scientists observe what the machines may be learning and then add new inputs and ideas to the system.

“A lot of times we forget that even though it’s big data, the amount of data that the machine has access to pales in comparison to the amount of data we’re absorbing and have access to,” he said. “We’re building intuitions and holistic pictures in our minds and we see these connections that the machine might not even have the possibility of seeing because it doesn’t have the right data.”

Humans have a powerful role in figuring out the sources of data to give the machine and projecting their intuition, he added.

Still, Timothy Estes, founder and CEO of Digital Reasoning, pointed out that there are three key areas in which machine bests man – and, over time, they could give rise to some interesting social and cultural questions.

Humans will never be able to consume the sheer amount of data machines can process (unless it’s with some “Ray Kurzweil-style” man and machine merging), humans weren’t designed to receive thousands of inputs at once, and we’re ill-equipped to create a unified model of knowledge across that scale of information and make judgements from it, Estes said.

Recognizing that, he said, he predicts a social debate between adopting a “Google”-like model to artificial intelligence, in which the machine simply tells you what to do next, and a software model, that assumes more human agency.

“I believe we’re going to see that [debate] play out in the next decade between the software-centric model – a personal empowerment model – and a collective model,” he said. “And that’s the Skynet problem… you get a computer with intentionality that has access to data and the next thing you know you’re looking for a robot coming back from the future.”

Check out <a href=”http://gigaom.com/2013/03/20/structuredata-2013-live-coverage/“>the rest of our Structure:Data 2013 coverage here</a>, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Connected world: the consumer technology revolution
• A near-term outlook for big data