Kevin Kelly, in “Better Than Human,” tells us how the future is going to go down. As we increasingly automate existing occupations, we create new jobs in order to instruct and direct those robots. We build robots to take over the instructional positions, and create new jobs that set parameters and develop feedback loops. We build new systems that are flexible and dynamic and create more new jobs — such as data scientists — to analyze and build models for these new systems. It is obvious that in such a world, where static models cannot keep up, data scientists will be indispensable.

Given this future, the argument presented in a recent GigaOM post, “We don’t need more data scientists – just make big data easier to use,” misses some key points. Its premise — that we need simple ways to deliver big data to business decision makers — is correct in that we need better tools, yet it misses an important distinction about who will use these tools. To complete author Scott Brave’s analogy of web content systems, data scientists are the designers and the content creators of today, not the software engineers or the IT bottleneck.

Every organization will need someone wearing the data scientist hat just like very organization has people responsible for product, sales, marketing and support. Unfortunately, to date, the tools available to data scientists have been rudimentary. Data scientists have had to learn diverse and complex computer languages for working with data. That world is changing as we create simpler ways for data scientists to use big data.

Even recommendations are more than just recommendations

Retail recommendations are a basic example of why we need data scientists today. A retailer sees a return on investment by showing any suggestions to its customers, whether or not the recommendations are relevant. That has been true ever since magazines and gum first flanked the checkout line. Data scientists can do better, but not by creating more sophisticated mechanisms for recommending the same gum and magazines. Recommendations are a very crude use of data.

As veteran retailers know, the art of cross-selling and upselling is not as simple as showing a customer what other customers bought or a list of related items. Data scientists are now building models that anticipate our needs. They are finding new ways to delight us and define experiences that create natural guides toward ideal outcomes. Soon, we will not need recommendations; with well-defined models, the subsequent decisions and actions will be obvious and then they will be automated.

Consider a retail experience that goes beyond recommendations. The elders among us recall once having delightful interactions with a local shopkeeper. The teller was the owner and knew every item in the store and every customer in town. The owner was able to personalize in his head, matching each customer to precisely what they needed. We lost this experience when we started to optimize for price and diversity over experience.

Using data analysis, we no longer need to choose between efficiencies and experiences. Instead, we expand the definition of efficiency to include customer experience. In order to stay competitive, a retailer can no longer focus just on the efficiency of moving products to the right locations. Competitive retailers are building a deep understanding of how customers use their products. Efficiency means matching the right product to the right customer at the right time, even as both evolve.

And that’s the easy part

Retail is a tangible example because we are all consumers, but retail is not the most instructive example of why data science is becoming such an important field. The fields of health care, education and energy are all evolving at a rapid pace:

• We are discovering new drugs and new techniques, and finding new ways to measure how to best care for patients.
• The ubiquity of laptops and tablets, the emergence of online courses and the thirst for ongoing education are creating new opportunities to measure (not test) how students learn and refine how we teach.
• Energy is on the cusp of a radical shift — from oil to gas, from fossil fuels to renewable energy, from manual heating, cooling, driving and flying to automated, fine-tuned and efficient use of the energy we produce.

We cannot begin to conceive of static models that will track and analyze these advances. We cannot replace data scientists with vertical defined, highly segmented solutions delivering slightly better analytics to existing decisions makers. Instead, we need to develop new creative thinkers and give them high-level tools to help them apply detailed data-driven models across a range of challenges.

Like storytellers, data scientists embody the heart and soul of an organization and find ways to make it better. Every organization is going to employ someone whose responsibility is to use data to drive automated decision systems. With time, the decisions these data scientists make will become obvious and can be automated. Today’s decision makers get to spend time on more important jobs we haven’t even thought of yet.

We need data scientists, and we need hundreds of thousands of them.  They will do their magic, create new ways of experiencing life, products and services and, as Kevin Kelly says, “dream up new work that matters.”

Omer joined WibiData in 2012 having worked for three years at Cloudera, bringing Hadoop to the enterprise market. He was previously at Vertica (now HP) where he led the Field Engineering team. Omer holds a B.Sc. in Computer Engineering from Tufts University and was a visiting scholar at Oxford University reading in Computation and Engineering, focusing on architectures for large scale distributed systems.

Feature image courtesy of Shutterstock user Minerva Studio.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Health care and big data in 2012
• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization

Across increasing verticals, it’s never been easier to use apps and services to instantly find things that might match my interests.

But I’ve been thinking about those on-demand recommendations in the context of a conversation I had this week with TastemakerX founder and CEO Marc Ruxin about the different between interests and taste. (His startup, which launched in June, gives music fans a stock market-like game for discovering and learning about artists and he was at the GROW Conference this week in Vancouver to give a talk about the value of taste.)

Taste, he said, cuts to the core of a person’s evolving identity – his past, present and future. Interests can be more casual and transactional – stuff you might have, not the person that you are.  But taste implies a commitment of time and thought and, beyond defining who you are, it can inspire others.

“You have the taste that you’ve done the due diligence to have,” he said.

In advocating for taste, he argued that in connecting people and spreading awareness about movies, food, films and cultural trends, social networking has made it so that “everything, everywhere looks the same.”

Even though the Web has given us the tools to find like-minded people, its filters aren’t good enough for us to always find and communicate with them at scale, he said.  The Internet has the potential to connect people with the most esoteric taste – such as those who love funny pictures of cats (a la this) – but not all tribes that organize around culture can find and deeply connect with their cohorts online.

As a result, his bet is that people will increasingly opt for vertical-specific communities that, in a sense, celebrate, organize around and help people develop taste.

“It isn’t about good taste or bad taste,” he said. “But that you make your taste your own.”

I’m not sure I totally agree with his assessment about the extent of social networking’s homogenizing effect, but I do think there is something to be said for dipping below the surface of what happens to be most accessible to really seek out and figure out the content and communities that most authentically connect with who we are.

Especially with the overwhelming amount of information available, it’s much easier to listen to the music Apple says I might like and read the books Amazon suggests, instead of investing the time and energy to develop my own taste — and I often follow Apple and Amazon’s recommendations, happily. But I also feel, a little bit, like I’m cheating. Putting in the work to make good choices from a larger pool of options isn’t just a learning and discovery process about the greater category of content, it’s a learning and discovery process about ourselves.

The web has brought many more options to our fingertips, making the process of selecting the communities and content we want to spend our time with all the more difficult. And, in some contexts, time-saving data-driven and human-curated discovery tools are incredibly valuable. But in others, I’m starting to think, I’d rather lose time than taste.

Disclosure: True Ventures is an investor in TastemakerX and the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.

(Image via Kram78 via Shutterstock.)

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How social discovery is transforming entertainment
• 12 tech leaders’ resolutions for 2012
• NewNet Q4: Platform mania and social commerce shakeout

Yes, we have Flixster and the like, but there we’re talking dense databases rather than properly personalized recommendations. And the fact that there’s no winner yet in this space – apart from intra-platform discovery engines such as Netflix’s – could mean one of two things: either that not many people actually want such a service, or that nobody’s got it right yet.

At least two Berlin startups hope the latter is true, but the approach each one is taking is very different. In the social-driven discovery corner we have Tweek.tv and then, as of yesterday, in the algorithm corner we have Foundd. Which idea works better?

“We use an algorithmic approach to calculate personal recommendations for each individual,” Foundd’s Lasse Clausen told me. “While there’s definitely the element that you want to see what your friends have seen, they’re generally not as good as algorithmic ones because if you test it, there’s a surprisingly small number of friends that really share your taste.”

Boom! Over to you, Tweek.tv:

“We believe there is nothing stronger than a recommendation from a person you trust and whose taste you know when it comes to movie recommendations,” co-founder Marcel Duee said. He noted that, although Reed Hastings owns the best algorithm in the world after the Netflix competition, when Facebook published the package of Social, Open and Interest Graph, Hastings said it trumped the algorithm. “We also we see that click-through rates for social recommendations are significantly higher than for pure algorithm recommendations,” he added.

Pow! There’s a valid point there — and Netflix has more than 900 engineers working on its recommendation algorithms. But then again, Foundd is cross-platform (it searches across both Netflix and iTunes). Back to Foundd’s Clausen:

“Netflix put a lot of effort into predicting how much I’ll like every one of their movies. But I don’t really care whether I’ll dislike a movie with 2.3 or with 2.35 stars, it doesn’t help me find a good movie. We focus on giving more accurate predictions over a smaller number of movies and those that you’ll really like. Also, Netflix doesn’t give recommendations for movies that several people will enjoy together.”

Clausen also pointed out that Foundd’s longer-term vision includes being a recommendation engine for a variety of content, including TV shows (like Tweek.tv) but also apps, games and books. “So you even if you only rated movies, you’ll be able to get recommendations for books or games on your iPad for example,” he explained.

So to sum up, one side thinks friends are the most useful arbiters of taste, while the other prefers to put its faith in the user’s own demonstrated taste (via a lengthy series of film ratings at sign-up). I can see advantages and problems with both approaches.

The social recommendation engine has the advantage of ease: simply plugging your friends into it tells the service what it needs to know, without the need for a questionnaire. But on the other hand, you may like your friends much more for who they are than for their tastes.

The algorithm approach is much more personal, which is a strong advantage, but it also requires a lot more work on the user’s part. Also, it’s really hard to take on algorithm beasts such as Netflix.

Still, this all brings us back to the issue of the market. I can’t help but feel skeptical about it: the idea has been around for a few years now, so shouldn’t at least one cross-platform recommendation engine have gotten big by now?

Could it be the case, especially with paid content, that people invest both their time and money in one or two platforms and are comfortable enough within those confines to not require third-party discovery?

If you ask me — and the corpses of services such as Sortflix and MyZeus — the jury’s still out on this space.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• How social discovery is transforming entertainment
• Connected world: the consumer technology revolution

The purchase isn’t too surprising. On the heels of its $84 million equity financing round in September, Opera Founder and CEO Arnab Gupta told me he was looking to go shopping for technology and for brainpower. Commendo, which sells a consumer recommendation engine called Recommendo, seems to fit both desires.

Aside from its obvious utility in e-commerce, the Commendo acquisition could help Opera establish a business around the video-streaming model it has developed internally. That model relies heavily on accurate recommendations among users with similar interests, as well as a peer-to-peer architecture, in its mission to improve both the volume and delivery of streaming video services. Commendo’s brains and the intellectual property underlying Recommendo could prove valuable in creating a custom recommendation engine for such a service.

During the Netflix Prize competition in 2009, Commendo employees participated as part of the winning team, named BellKor’s Pragmatic Chaos, while Opera employees were part of runner-up team The Ensemble. The goal was to improve the accuracy of Netflix’s recommendation algorithm by 10 percent, which both teams did. They actually finished in a statistical tie at 10.06 percent, but BellKor’s Pragmatic Chaos submitted its final algorithm 20 minutes earlier than did The Ensemble.

Watch the video below to see my on-stage discussion with Opera’s Gupta at our Structure: Data event last week.

Image courtesy of Flickr user Bitterjug.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• CES 2012: a recap and analysis
• Who and what to watch in the new era of the living room

Clever Sense uses a combination of machine learning and artificial intelligence to help best predict which restaurants, bars and clubs you’d like to go to. I’ve used it on the iPhoe and it’s a pretty smart tool, that takes input from a user and turns it into better and better results. Clever Sense will join Google local team and will help improve Google’s ability to suggest places for people to discover.

Babak Pahlavan, co-founder and CEO of Clever Sense, said combining with Google will help accelerate the start-up’s plans to curate the world and brings its discovery service to many more people. It sounds like Google will continue to let Alfred live on, but it’s also possible we could see it shuttered at some point with its features subsumed into Google properties.

This continues a buying spree for Google, which bought local reviews service Zagat in September and local deals aggregator The DealMap in August. Google also recently launched Google Schemer, a recommendation engine for activities, and it’s moving forward with its Google Wallet and Google Offers products, which tie payments and deals together. Stepping back, it’s clear Google wants to be the lens through which people see the local world around them and the layer between consumers and local merchants. By putting together reviews, content, recommendations and deals, it can draw in consumers who are increasingly using mobile devices to search for local results and explore the world around them, and it can pitch itself as an advertising channel and customer acquisition tool for merchants, which are moving away from old paper coupons and print ads.

Clever Sense also gives Google more artificial intelligence as it competes with Apple and its new voice assistant Siri. While Siri is still off to an uneven start, it shows a lot of promise in understanding consumer intent and eliminating barriers that keep people from information. Google itself is trying to become one big artificial intelligence product but it can still use help in becoming more aware of people’s interests and intent. EBay just bought Hunch, another recommendation engine, following up on an earlier acquisition of WHERE. Not every recommendation is going to succeed or get bought up, (see Bizzy) but there’s clearly an interest by some big companies to get smarter and provide very intelligent assistants and robots that understand what people want.

Marissa Mayer, Google’s head of mobile and geolocation, talked at South by Southwest about using mobile phones to help bridge the physical and digital worlds and help bring the “power of here” to reality. And more recently, she talked about providing more serendipitous suggestions to users. More and more, we’re seeing that local vision take shape as Google buys up the pieces to bring it all to life.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Siri: Say hello to the coming “invisible interface”
• Analyzing the wearable computing market
• 12 tech leaders’ resolutions for 2012

When you go to Gruvi.tv’s Facebook app, it will recommend films to you based on likes and interests you’ve submitted, as well as those of your friends. By doing so, it can provide a more personalized view of which movies users might prefer, which might not match what’s popular or what’s currently out in theaters. As users interact with the app — for instance, by rating films — it automatically updates their recommendations to match that new data. That gives everyone a different personalized landing page whenever they come to the app.

But Gruv.tv is more than just a recommendations engine. On the front end, Gruvi.tv provides Facebook users a personalized way to determine which movie might be best suited for them, but on the back end, it’s designed to give movie marketers tools to help capture audiences and get them into theaters. Gruvi.tv’s platform and gives them the ability to offer user showtimes, show users which of their friends liked a certain film, as well as add ticketing, quizzes and promotions to the page.

Gruvi.tv also sees an opportunity in helping studios to target new films to moviegoers after a campaign is over. Typically, once a film is launched in theaters, the studio loses its relationship with moviegoers on the Facebook page as they stop visiting, but Gruvi.tv believes it can provide tools to transition fans of one film to view another that may be in theaters or coming soon. And finally, the company can provide detailed analytics about user engagement around the fans who check out movies using its system.

Studios increasingly recognize the importance of social networks for promoting their movies before they are made available in theaters, and are spending big to reach audiences on Facebook. Gruvi.tv estimates that studio spend on Facebook marketing has gone from zero to €300 million ($428.6 million USD) over the last 18 months, and the average marketing budget per film has risen by about 50 percent over the past year. Gruv.tv hopes to tap into those dollars by being the funnel between its recommendation engine and film fan pages.

Since being launched in beta in Denmark, Gruvi.tv has worked with studios such as Warner Bros., 20th Century Fox and Sony Pictures to boost viewership with its tools. The app is now stepping into the U.K. with a campaign for the launch of Sony’s Bad Teacher in that market.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Here Come the Social TV Apps
• How consumer media will change in 2013
• What the shift to the cloud means for the future EPG

So how does it work? Dixon told me via email that Hunch uses preferences from existing users to predict recommendations for users not known to the system. “Basically we have about one million people who have Twitter (or) Facebook connected into Hunch and told us about their music preferences,” he said. “We use this ‘known’ data to make inferences about unknown users based on who they follow, who follows them, and other signals.”

Dixon also emphasized that Forage is exploratory by nature — it won’t necessarily list your favorite bands, but rather music you may want to try. And I gotta say, it actually works pretty well for that purpose. Not only does it serve up an interesting selection of videos, users can also link to the resulting playlists. Unfortunately, it struggles with one of the issues many music-focused YouTube mashups are facing: Some major labels aren’t allowing third party sites to embed their clips.

Don’t expect Dixon to invest a lot of time into resolving this issue. “Forage is really just a little hack I did,” he told me said, adding: “And I’m not a great programmer.” Still, it’s an interesting little demo of the power of Hunch and its take on taste-based predictions, which in turn could be really useful for many online video platforms and services.

Recent research from the BBC has shown that content recommendations have a significant impact on the time people spend watching video online, but companies like Netflix have in the past struggled to add social elements to these recommendations. Hunch’s approach of using known patterns to predict unknown preferences could help to make these types of recommendations much more valuable.

Chris Dixon stopped by our office a few months ago to talk about his experience as an angel investor and his thoughts on the VC model. Check out the first part of the interview embedded below, and read this post for the second part:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Controversy, courtrooms and the cloud in Q1
• 12 tech leaders’ resolutions for 2012

MediaUnbound has helped to build static and dynamic personalization and recommendation engines for a number of clients, including Napster, eMusic and MTV Networks. The company has focused primarily on the music industry, but Corey Ferengul, Rovi’s executive vice president of product management and marketing, said in a phone interview that the same software can be used to provide recommendations for other media types that Rovi has in its database, including movies, TV, games — even books. In fact, he says that many of MediaUnbound’s customers use its recommendation system along with Rovi’s data products.

The first step for Rovi will be packaging MediaUnbound’s recommendation technology to customers that buy its media data. But the company could also integrate the technology into its TotalGuide digital programming guide, which could enable it to extend more personalized movie and TV show recommendations to end users.

Rovi evaluated a few different recommendation services before acquiring MediaUnbound, and according to Ferengul, it was just as interested in acquiring the company’s talent as its technology. “Recommendation is a hot space right now, but there isn’t a single answer — there will be continual answers that need to be developed over time, and we needed a good team to keep us looking forward,” he said.

Cambridge, Mass-based MediaUnbound was founded by three Harvard University students in 2000. Privately held, the company has been profitable since 2003.

Related GigaOM Pro content:

The Paradox of Thinking Outside the (Set-Top) Box (subscription required)

Whiwa is run by two developers out of the Czech Republic. The duo told me they have a lot of plans for their platform, but at this point it’s clearly still very experimental. However, the idea behind it is certainly interesting, and it begs the question: Could a movie site like this eventually become as big as Last.fm? Or is video just too different of a beast than music?

Whiwa.net is a movie community site that works a little bit like the original Audioscrobbler service that eventually became the technical core of Last.fm: Users can download a small client that tracks which videos they’re watching on their PC. This data is synchronized with the Whiwa.net database and added to a user’s profile. Users can also discover new content by browsing the profiles of others with similar taste.

I played around with Whiwa over the last few days, and it does work. Well, sort of. The database is still fairly small, and the client doesn’t play well with all media players. No VLC for example, and no, no OS X, either — Whiwa is currently Windows only.

Whiwa’s two developers tried fingerprinting and other more advanced content recognition methods, they told me, but those caused too many errors, so they eventually settled for the fastest and easiest approach: The client simply checks for the file or disk names and cross-references this information with IMDB data. Users can also manually add movies they’ve seen in the theater or on TV. They plan to add a more advanced recommendation engine and a Facebook application in the near future.

I found paying with Whiwa fun, if only for the fact that it made me think about the viability of such a model in the online video space. Most of us consume far fewer movies than songs, and TV shows require a different kind of commitment than music. Fall in love with a show’s pilot, and you’ll be busy watching all of the following episodes during the rest of the season and thus far less interested in dozens of recommendations for other shows.

Then there’s the legal side. Last.fm recently got into hot water when TechCrunch alleged that the site’s owner CBS handed over the personal information of thousands of listeners to the RIAA. Some have doubted whether this type of data would really be useful to the music industry. After all, how would a label know whether a Last.fm user got a recording as an illegal download or through legal means? Movie data, on the other hand, would be much less ambiguous. There’s simply no way to legally watch a Hollywood blockbuster on your home PC right after its theatrical release.

However, there’s also a lot to like about the idea of a Last.fm for movies. People like to share what they’re watching, and previous efforts to capture this type of data have proven either too complicated or inconclusive. You can tell your friends about your favorite movies on Facebook or try to find new online videos through a recommendation service like ffwd, but those things usually require a lot of explicit action on your part. You have to enter movie titles, vote for clips, etc. Moreover, offering up good recommendations based on those votes can be really, really hard, as Netflix well knows.

The beauty of the Last.fm model is that you just do what you’ve always done — watch a movie or listen to a song — and the service monitors and calculates your taste automatically. Of course, doing that for movies and TV shows would really only well work if it captured the vast majority of our video consumption. This means it would have to track our viewing habits online, on the tube and even on cell phones. It’s not impossible — but something tells me that we’ll still have to wait a little while before someone is going to become the Last.fm of newteevee.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• 10 ways big data changes everything
• 12 tech leaders’ resolutions for 2012