The way it works, said Mortar Co-founder and CEO K Young, is that his company will choose eight companies (in addition to the two it has been working with already) to implement custom systems based on their specific needs and businesses. Mason, Conway and Shron will split their time among the 10 total companies, but will be much more than advisers — they’ll actually dig into the data and work hands-on to ensure the right techniques and algorithms are applied in the right places.

The applicant companies will keep any custom code, but the ultimate goal from Mortar’s perspective is to learn some best practices and create reusable building blocks that will let anyone create recommendation engines without pre-existing data science knowledge. Recommendation engines are commonplace on large web sites (Netflix, Spotify, iTunes, Google, Amazon, LinkedIn, Eventbrite and the list goes on) but smaller companies can sometimes struggle to do them, or to do them well. Young hopes Mortar can establish an open source reference architecture of sorts that makes it easy to implement everything from building data pipelines to the actual algorithms that power recommendations.

“They’re really common and they’re really useful, but they’re really hard,” he said. “That’s why [a reference implementation] hasn’t been done before.”

They can get pretty complex, as evidence by this Netflix example.

Presently, Young explained, anyone wanting to build a recommendation system probably knows some of the algorithms to begin with and then gets to work researching how to implement them with specific processing frameworks (e.g., MapReduce) and on their specific data. Alternatively, they might have to hire a consultant that helps them build the recommendation engine. Either way, he noted, they’re probably not open sourcing it at the end because it’s presumed too valuable a competitive edge.

Mortar Data’s recommendation framework will be based on Pig, Python and Java, just like the company’s flagship platform for creating Hadoop jobs. Those languages will make the implementation more accessible and customizable by more people, Young said.

Really, he added, any web site or service that has multiple customers and deals with multiple entities — be they restaurants, songs, dating profiles, artisan necklaces, what have you — should have some sort of recommendation engine to help provide a more-intelligent customer experience. “It should become so ubiquitous that any service you go to knows enough about you to put forward the things you actually want to see,” Young said.

There is, however, one catch to Mortar’s plans as they stand: Because the service is hosted on Amazon Web Services, anyone interested in having Mason, Conway, Shron and Mortar work on their systems must have their data in AWS or be able to move it there. The initial reference implementation will likely be AWS-centric, too, but Young hopes contributors will use it and share methods for running it atop other platforms.

Feature image of Hilary Mason at Structure: Data 2011 courtesy of Pinar Ozger (www.pinarozger.com).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

“Bitly is announcing today that Peter Stern has resigned to pursue other interests. “Peter has been a key leader and contributor to the Company,” said Bitly Board member Sam Mandel. “In particular he has been instrumental in transforming Bitly into a successful business while growing its unparalleled data set. We are very happy that he will remain a shareholder and supporter.””

Stern was previously the founder and CEO of Zenbe, a mobile and web-based collaboration service that was acquired for talent by Facebook in 2010.

Bit.ly released a new API in January to allow companies to track the flow of news and information on Twitter and across the web, and has been focused on providing more real-time analysis. We’ll be talking about some of the challenges associated with real-time data at our Structure Data conference next week in New York City.

“Bit.ly is in a great position to see this kind of social data. We see a slice of what people are sharing and reading across the social web. So this is a way to start to put that power in the hands of people who can build interesting things with it,”  Hilary Mason, Bit.ly’s chief scientist, said in an interview in January.

Bit.ly is a company that grew out of Betaworks, and it shortened 8.4 billion links in 2012. Bit.ly raised $15 million led by Khosla Ventures this summer and has rasied more than $28 million in total.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Frenemy mine: The pros and cons of social partnerships for online media companies
• The state of cross-platform media measurement
• Social-TV apps and consumer behavior

“The idea is that people will start to build real-time search into their applications,” said Hilary Mason, Bit.ly’s chief scientist, in an interview explaining the release. The company announced full details of the API in a blog post on its website Tuesday.

The company launched its real-time product to users this summer. It is available to users with Bit.ly accounts at rt.ly, where they can fill out various information fields to see which news stories are getting the most hits across the globe, or see which topics are popular. For instance, this is where you could search for food articles in Brooklyn, New York, United States, and see that something called a “Turkish bagel” is riling people up.

Now, with the release of this API, developers will be able to integrate the real-time search features into their apps, showing what information sent through Bit.ly is being read, where it’s being read, and how often. For instance, a publisher could show which of its articles are the most popular in a particular geographic area, or companies could track where around the globe consumers are reading about its product. Bit.ly’s real-time product also allows for the tracking of particular topics or phrases that are getting a good deal of attention via social media.

Of course, all of the tracked trends on what’s being read right now are necessarily stories tweeted from Bit.ly links, but that’s no insignificant measure. The company, which came from Betaworks, shortened 8.4 billion links in 2012. The company raised $15 million led by Khosla Ventures this summer, bringing its total funding to more than $28 million.

“Bit.ly is in a great position to see this kind of social data. We see a slice of what people are sharing and reading across the social web. So this is a way to start to put that power in the hands of people who can build interesting things with it,” she said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Sector RoadMap: Content personalization in 2013
• Frenemy mine: The pros and cons of social partnerships for online media companies
• The state of cross-platform media measurement

In the next couple of months, URL shortening service bitly is launching a real-time search engine that would let publishers see how their content — including books on Amazon — is being shared. And TOC’s Joe Wikert thinks publishers should track data like how far readers get in an e-book before they stop reading. “Amazon is probably gathering this data already,” he said. “At some point publishers will probably be able to buy access to that information,” but it’s even better if they can collect it themselves.

Audience member Dominique Raccah, publisher of the innovative Sourcebooks — whose e-book revenues were up 795 percent in 2011 — said publishers should use data to answer questions like how many of their books fail and how much it costs them to publish a book.

Sourcebooks is collecting data about their books and about how readers use their website. The biggest success they’ve had is using data to position books, Raccah said — “where we think the book is about one thing, and the consumer thinks the book is about something else.” Publishers can poll readers to find “what’s exciting somebody, what’s really turning them on — what it is that moves the needle for them.”

Bitly’s Mason agreed that data doesn’t have to be big or “require a huge amount of infrastructure” to be useful. And publishers can collect data through resources that weren’t explicitly designed for that purpose. For example, Mason said, if a publisher can’t decide what a book’s title should be, they should buy Google Ads for two different titles and see which title gets more clicks. The product you’re advertising, she said, “doesn’t actually have to exist.”

Correction: An earlier version of this story incorrectly stated Hilary Mason’s last name. Her last name is Mason, not Rosen. I apologize for the error.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How big data analytics drives competitive advantage
• How data warehousing is now a cost-effective solution for businesses
• How to manage big data without breaking the bank

I was reminded of New York’s role in the emerging big data ecosystem over the weekend, when I was catching up with my weekly reading.

• eBay, which had acquired Hunch in November 2011, is now building out its operations in New York with a new 35,000 square-foot office, Ryan Kim reported last week. ”When fully built out, a majority of the workers will be developers, data scientists and statisticians,” he added.
• Microsoft has hired a bunch of former Yahoo researchers in New York.

Steve Lohr of The New York Times wrote about Microsoft:

The group’s research focuses in large part on the application of advanced computing tools to the social sciences. It is a fast-growing field fueled by the vast new data sources of the Web, social-network communications and from sensor-equipped devices like smartphones. The potential is enormous, as Google and Facebook prove. But Microsoft has trailed so far.

Beyond these two recent announcements, there are several other startups that are experimenting with data. Take URL shortener, Bit.ly, for example. The service, which is based in Manhattan, is creating new news reader experiences based on the data it collects and from the social context that data carries. Businessweek writes:

This year, Bitly is introducing a suite of data products for professionals developed in part by Mason and her team of six scientists and engineers. One, dubbed Bitly Realtime, tracks terms that receive sudden bursts of attention. Another is a reputation-monitoring system. The goal of the products is “to give people a Spidey sense about what’s going on on the Internet that’s relevant to them,” says Mason.

Hilary Mason, who is the chief scientist at Bit.ly told me that in “New York we are more interested in telling stories” from of data as opposed to how “big” the data is, or what database technology you happen to use, Mason told us. “This is how businesses, marketers, and social scientists need to think about data to make rational decisions.”

New York has a long history of learning from data, thanks to the quantitive revolution that swept Wall Street. Financial services, was the first real big data vertical, and quants were Wall Street’s data scientists. The markets for tradable instruments, high-volume and high-velocity data streams all came from Wall Street. (Actually, that was the rationale behind why we host our Structure: Data conference in New York.)

Mason believes that New York can leverage big data to its advantage. From art to fashion to media, New York has enough creative talent to be able to ask the right questions from the data. A good example is the Explore feature on Foursquare, which co-founder Dennis Crowley calls the “big data driven recommendation engine for the real world.” (Here is a presentation of technology behind Explore that is pretty cool.)

Similar startups such as Knod.es are helping make sense of social data as well. For instance, the startup sucks in your social data across platforms and let you ask questions like “Who do I know who likes cheeseburgers in Paris?”

“There are a bunch of interesting things happening in NYC, not all of which are startups,” Mason says, singling out Wes McKinney’s the Pandas Project, which is a time-series analysis system drawn from his experience in finance. Mason calls it “awesome.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• The state of cross-platform media measurement
• Social third-quarter 2012: analysis and outlook