The way it works, said Mortar Co-founder and CEO K Young, is that his company will choose eight companies (in addition to the two it has been working with already) to implement custom systems based on their specific needs and businesses. Mason, Conway and Shron will split their time among the 10 total companies, but will be much more than advisers — they’ll actually dig into the data and work hands-on to ensure the right techniques and algorithms are applied in the right places.

The applicant companies will keep any custom code, but the ultimate goal from Mortar’s perspective is to learn some best practices and create reusable building blocks that will let anyone create recommendation engines without pre-existing data science knowledge. Recommendation engines are commonplace on large web sites (Netflix, Spotify, iTunes, Google, Amazon, LinkedIn, Eventbrite and the list goes on) but smaller companies can sometimes struggle to do them, or to do them well. Young hopes Mortar can establish an open source reference architecture of sorts that makes it easy to implement everything from building data pipelines to the actual algorithms that power recommendations.

“They’re really common and they’re really useful, but they’re really hard,” he said. “That’s why [a reference implementation] hasn’t been done before.”

They can get pretty complex, as evidence by this Netflix example.

Presently, Young explained, anyone wanting to build a recommendation system probably knows some of the algorithms to begin with and then gets to work researching how to implement them with specific processing frameworks (e.g., MapReduce) and on their specific data. Alternatively, they might have to hire a consultant that helps them build the recommendation engine. Either way, he noted, they’re probably not open sourcing it at the end because it’s presumed too valuable a competitive edge.

Mortar Data’s recommendation framework will be based on Pig, Python and Java, just like the company’s flagship platform for creating Hadoop jobs. Those languages will make the implementation more accessible and customizable by more people, Young said.

Really, he added, any web site or service that has multiple customers and deals with multiple entities — be they restaurants, songs, dating profiles, artisan necklaces, what have you — should have some sort of recommendation engine to help provide a more-intelligent customer experience. “It should become so ubiquitous that any service you go to knows enough about you to put forward the things you actually want to see,” Young said.

There is, however, one catch to Mortar’s plans as they stand: Because the service is hosted on Amazon Web Services, anyone interested in having Mason, Conway, Shron and Mortar work on their systems must have their data in AWS or be able to move it there. The initial reference implementation will likely be AWS-centric, too, but Young hopes contributors will use it and share methods for running it atop other platforms.

Feature image of Hilary Mason at Structure: Data 2011 courtesy of Pinar Ozger (www.pinarozger.com).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Aiming to figure out why America is becoming more politically polarized despite traditional beliefs that societies naturally move toward the middle, a group of Stanford researchers considered how our natural biases affect the way we interpret information. What they found is that people tend to view the world through red- or blue-colored glasses: when we see inconclusive information, we intepret it in ways that support our natural political biases and ignore the aspects that don’t. So if you show the exact same piece of inconclusive information to a group of people, it will likely lead to more polarization rather than to general consensus on the meaning.

It turns out, this phenomenon extends beyond clearly biased media such as FOX or MSNBC and into more objective content sources on the web. When the researchers applied their model to online recommendation engines, they found that pieces of content most-relevant to users are “always polarizing,” whereas pieces of information that are merely similar to something someone already likes are only polarizing if the person is already biased. In short: While they’re able to ignore or at least view objectively less-important stuff, even pretty middle-of-the-road people will take a hard stance on stuff that matters to them.

Of course, how one reacts to research like this largely depends on what one is trying to accomplish. The researchers involved appear to be all about moving people toward the middle on some issues, which is why they created a federal-budget app called Widescope that lets people configure their own budgets and then shows them the similarities with the various budget proposals floating around Washington, D.C. They’ve also looked into creating social systems that counteract polarization by using trusted information sources (a press release explaining the research suggests Rush Limbaugh or Rachel Maddow) to present information that biased individuals might otherwise be inclined to dismiss.

Applied generally to the web, this approach might help mitigate some of the effects of the hyper-personalized experience that’s now possible. You know, the kind of thing that happens when you fill up RSS readers with sources you like, follow like-minded people on Twitter, and  sign up for services that use machine learning to surface even more of the same content based on that homogeneous reading activity. Or when you keep searching for the same stuff on Amazon or viewing the same types of movies on Netflix.

Services that go beyond “injecting serendipity” into their content feeds could actually try to broaden users’ minds by surfacing content that’s in some ways very different or counterintutive to what a simple interest graph might show. I’m not sure how this would look algorithmically, but I’m envisioning, for example, a semi-regular insertion of content from sources or genres considered the opposite of a readers’ norms but that touch upon topics they’re interested in. Or vice versa.

I genuinely believe most web startups trying to tackle the problem of content curation want to be helpful as possible, are aware of issues such as biased assimilation and are at least considering methods for counteracting it in order to give users a broader view beyond just what those users think they want to see.

On the other hand, if you wanna lock people into their current beliefs or their current content-consumption habits, that’s probably a lot easier to do. And sadly, for some politicians and special interest groups, that probably suits them just fine.

Feature image courtesy of Shutterstock user Kutlayev Dmitry.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How energy data will impact the smart grid
• Over the top in 2012: trends and technologies to watch
• The importance of putting the U and I in visualization

The trick to doing recommendations right, according to Eventbrite Director of Data Engineering Vipul Sharma (who will be speaking at our Structure:Data event in March), is rooted in both business and data architecture. Companies must know who their audience is, what types of data they can collect and how they can best use that data to discern what consumers really want. Or, to put it more succinctly, companies have to understand data science.

The Amazon microcosm

Amazon is a good example of the approaches a company might take. The e-commerce giant, Sharma explained, used to use a hierarchical model to recommend additional purchases to shoppers. Products were indexed in such a way that, for example, the system would always recommend batteries to someone buying a camera. However, as the product catalog grew and Amazon picked up its analytical abilities, it moved to the current model of recommending purchases based on what other people who bought the same thing also bought.

Now, Amazon is able to present more accurate suggestions because it’s using real-world purchase data instead of static indexes that make assumptions about what someone should buy. Presenting this information as “Customers Who Bought This Item Also Bought” is not only a way to sell more stuff, it also provides shoppers the peace of mind of knowing they’re pairing items that — if the wisdom of the crowd is to be believed — go well together.

Of course, Amazon also uses a process called collaborative filtering to make recommendations to members even before they start looking at individual items. These recommendations aren’t based strictly on products that are frequently bought together, but also on how shoppers with similar purchase histories and interests tend to behave. Amazon actually explains the process pretty clearly on its Recommendations FAQ page:

We determine your interests by examining the items you’ve purchased, items you’ve told us you own items you’ve rated, and items you’ve told us you like. We then compare your activity on our site with that of other customers, and using this comparison, are able to recommend other items that may interest you.

Down, I like. Otep, not so much.

But Amazon does have one trick up its sleeve that many other companies don’t: Shoppers spend a lot of time on its site and many of them might actually be willing to put in a little effort to get more-accurate recommendations. So, Amazon is able to build even better recommendations for users by asking them questions about recommended items — do they own it, do they like it, and is the purchase that spurred the recommendation worth using for future recommendations. The better the information a company has about what users actually want, the better recommendations (or ads) it can show them.

Everyone does it different

However, Sharma explained, every web company has its own unique methods for making recommendations. Facebook, for example, relies on users’ social graphs to make recommendations based on what someone’s friends also like (it also relies on users’ stated interests primarily to serve ads). It’s a platform built on the idea of connecting with friends, so it assumes users care what their friends are up to and are interested in.

Netflix takes a different approach (its Facebook Connect feature notwithstanding), focusing its recommendation efforts around items. Its algorithms are about calculating the relationships among pieces of content based on factors such as genre, actors, ratings and even the sequence in which users typically watch stuff. Personalization in this case is more interest graph than social graph — Netflix knows what you like (or at least what you’ve watched) and suggests new content that’s somehow similar to it or related to it. As I explain here, a graph is essentially a method for storing data based on their relation to each other.

Source: Netflix

(Some, however, have suggested that clustering users based on their interests first might make content recommendations more accurate, because, they argue, what others with the same interests are watching is probably more relevant than how the content itself is related.)

In June, my colleague Stacey Higginbotham wrote about the mysteries behind Apple’s Genius recommendations in iTunes. The methods actually don’t seem too out of the ordinary, but a Quora post explaining them did get pulled …

Eventbrite’s “implicit social graph”

In order to accurately recommend events for its members to attend, Eventbrite has to use a combination of all these tactics. Its model takes into account what events someone has attended in order to discern and graph their interests, Sharma explained, but also builds what he calls an “implicit social graph.” The people in this graph aren’t necessarily a user’s friends, but rather are people who frequently attend the same conferences.

“We internally treat them as your friends and believe they’re part of a social graph because you share an interest with them,” he said. For Eventbrite’s purposes of recommending new events a user might like, if someone is always attending country music festivals or big data conferences, his peers are the people attending those events.

Eventbrite actually does use Facebook Connect, Sharma noted, but clarified that “it’s useful, but it’s not really great for us.” People don’t update their interests too often, he explained, and someone’s personal interests don’t necessarily align with their professional interests. And although Sharma didn’t mention it, others in the business of building interest graphs have noted that the interests people express publicly in front of our friends and family don’t always align with the interests we express by acting in certain ways.

Another wrinkle for companies, such as Eventbrite, that are more services than platforms is that users probably aren’t willing to stick around and answer questions to help the sites build better models. Amazon and Netflix users might provide express feedback on recommendations, and Sharma noted that even Facebook can assume users are less interested in particular friends when they block certain updates or actions from them. Because users aren’t addicted, he said (or perhaps because they don’t attach the same importance to events as to the movies they watch), Eventbrite’s approach to learning what users want has to be frictionless.

I’m not a member, so Eventbrite can only assume I want local events

And while the talk today is all about social graphs and interest graphs, Sharma thinks the future of online suggestions is object graphs. No longer will it be good enough just to know a user is into music, but companies will also have to know what instrument she plays. This will be more difficult for companies that can’t simply ask users about their specific interests, but Sharma said it’s now so easy (relatively speaking) to collect lots of data both internally and from other web services, and analyze it deeply, that this level of specificity should be possible even for small data science teams.

Personalization, after all, is a popular method by which web services prove their worth. Netflix claims that 75 percent of what people watch comes from some sort of a recommendation. As for Eventbrite, Sharma said, “We are not a subscription product, but when people love the product, they stay with the product.”

Feature image courtesy of Shutterstock user Ivelin Radkov.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010

It’s a solid update worth checking out if you’re interested in an app that helps you to find shows and movies on live TV as well as streaming services like Amazon Instant and Netflix – but the thing that I took away from a conversation I had this week with Dijit CEO Jeremy Toeman was something else. I asked Toeman why there’s even a need for an app like NextGuide, now that everyone from Google to Roku offers universal search across a multitude of services. His answer: Because we’re Switzerland.

NextGuide’s newest update includes a social stream that shows which content your contacts like – even if they’re not using the app.

Toeman argued that most companies involved in this space have too much of a vested interest to be good at becoming TV’s next recommendation engine. Subscription services like Netflix exclusively focus on their own offerings, and all but ignore anything on live TV. Cable companies on the other hand recommend both live TV and their own on-demand services like Xfinity.com – but they still don’t link to their competition. “I don’t see the Xfinity app searching Netflix for me,” Toeman quipped.

Device makers are in a bit of a different position, because they often want to present you with as many content choices as possible. But Roku exclusively focuses on viewing content that works on its own devices. That means you’re screwed if you want to watch anything on the TV in your den that doesn’t have a Roku, or even continue watching a movie on your iPad during your morning commute.

NextGuide on the other hand doesn’t care about any of that. The app supports content from 54 5 different content sources, including Netflix, Amazon and Hulu Plus. It currently puts a heavy focus on watching content on the iPad, but it can already be used in conjunction with a DirecTV DVR, and Toeman said that support for other devices and platforms is coming soon.

NextGuide extends this Switzerland-like approach to the way it handles social signals. The newest version of NextGuide integrates an activity stream that lets you see what your friends are doing on the service. But it doesn’t stop there: Users can also see what kind of shows their friends are liking on Facebook, even if those friends don’t use NextGuide themselves.

Missing from the app is the kind of restless conversation through pulled in Twitter or Facebook comment streams many others in the space are banking on. Toeman told me that he wants to use social networks to find recommendations for new content, not to relay what everyone is saying about shows and movies. “I don’t care what people tweet,” he told me.

Image courtesy of (CC-BY-SA) Flickr user twicepix.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Managing infinite choice: the new era of TV user interfaces
• Connected consumer third-quarter 2012
• Who and what to watch in the new era of the living room

Yes, we have Flixster and the like, but there we’re talking dense databases rather than properly personalized recommendations. And the fact that there’s no winner yet in this space – apart from intra-platform discovery engines such as Netflix’s – could mean one of two things: either that not many people actually want such a service, or that nobody’s got it right yet.

At least two Berlin startups hope the latter is true, but the approach each one is taking is very different. In the social-driven discovery corner we have Tweek.tv and then, as of yesterday, in the algorithm corner we have Foundd. Which idea works better?

“We use an algorithmic approach to calculate personal recommendations for each individual,” Foundd’s Lasse Clausen told me. “While there’s definitely the element that you want to see what your friends have seen, they’re generally not as good as algorithmic ones because if you test it, there’s a surprisingly small number of friends that really share your taste.”

Boom! Over to you, Tweek.tv:

“We believe there is nothing stronger than a recommendation from a person you trust and whose taste you know when it comes to movie recommendations,” co-founder Marcel Duee said. He noted that, although Reed Hastings owns the best algorithm in the world after the Netflix competition, when Facebook published the package of Social, Open and Interest Graph, Hastings said it trumped the algorithm. “We also we see that click-through rates for social recommendations are significantly higher than for pure algorithm recommendations,” he added.

Pow! There’s a valid point there — and Netflix has more than 900 engineers working on its recommendation algorithms. But then again, Foundd is cross-platform (it searches across both Netflix and iTunes). Back to Foundd’s Clausen:

“Netflix put a lot of effort into predicting how much I’ll like every one of their movies. But I don’t really care whether I’ll dislike a movie with 2.3 or with 2.35 stars, it doesn’t help me find a good movie. We focus on giving more accurate predictions over a smaller number of movies and those that you’ll really like. Also, Netflix doesn’t give recommendations for movies that several people will enjoy together.”

Clausen also pointed out that Foundd’s longer-term vision includes being a recommendation engine for a variety of content, including TV shows (like Tweek.tv) but also apps, games and books. “So you even if you only rated movies, you’ll be able to get recommendations for books or games on your iPad for example,” he explained.

So to sum up, one side thinks friends are the most useful arbiters of taste, while the other prefers to put its faith in the user’s own demonstrated taste (via a lengthy series of film ratings at sign-up). I can see advantages and problems with both approaches.

The social recommendation engine has the advantage of ease: simply plugging your friends into it tells the service what it needs to know, without the need for a questionnaire. But on the other hand, you may like your friends much more for who they are than for their tastes.

The algorithm approach is much more personal, which is a strong advantage, but it also requires a lot more work on the user’s part. Also, it’s really hard to take on algorithm beasts such as Netflix.

Still, this all brings us back to the issue of the market. I can’t help but feel skeptical about it: the idea has been around for a few years now, so shouldn’t at least one cross-platform recommendation engine have gotten big by now?

Could it be the case, especially with paid content, that people invest both their time and money in one or two platforms and are comfortable enough within those confines to not require third-party discovery?

If you ask me — and the corpses of services such as Sortflix and MyZeus — the jury’s still out on this space.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• How social discovery is transforming entertainment
• Connected world: the consumer technology revolution