Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Takeaways from the second quarter in cloud and data
• Amazon’s DynamoDB: rattling the cloud market

The purchase isn’t too surprising. On the heels of its $84 million equity financing round in September, Opera Founder and CEO Arnab Gupta told me he was looking to go shopping for technology and for brainpower. Commendo, which sells a consumer recommendation engine called Recommendo, seems to fit both desires.

Aside from its obvious utility in e-commerce, the Commendo acquisition could help Opera establish a business around the video-streaming model it has developed internally. That model relies heavily on accurate recommendations among users with similar interests, as well as a peer-to-peer architecture, in its mission to improve both the volume and delivery of streaming video services. Commendo’s brains and the intellectual property underlying Recommendo could prove valuable in creating a custom recommendation engine for such a service.

During the Netflix Prize competition in 2009, Commendo employees participated as part of the winning team, named BellKor’s Pragmatic Chaos, while Opera employees were part of runner-up team The Ensemble. The goal was to improve the accuracy of Netflix’s recommendation algorithm by 10 percent, which both teams did. They actually finished in a statistical tie at 10.06 percent, but BellKor’s Pragmatic Chaos submitted its final algorithm 20 minutes earlier than did The Ensemble.

Watch the video below to see my on-stage discussion with Opera’s Gupta at our Structure: Data event last week.

Image courtesy of Flickr user Bitterjug.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

Thanks to a combination of peer-to-peer networking and big data algortihms, cable or satellite providers could significantly reduce the costs of serving content while also improving the accuracy of personalized recommendation — and it relies on the omnipresence of set-top boxes.

Opera employees working under the team name Media Moguls developed the model as part of the company’s annual Opera Open challenge, in which employees compete to create the best product that utilizes the company’s analytics technology. Media Moguls won the competition.

How it works

According to team member Carmine Mangione, chief software architect at Opera, the problem facing most providers of streaming video is one of choice vs costs. While they might want to provide the largest libraries possible, that’s a very expensive proposition. In order to provide an effective streaming experience and provide features such as rewinding, Mangione explained, content needs to be served from memory rather than on slower, but cheaper, hard disks. On average, a single server could effectively serve about 100 people watching the same movie at the same time.

That’s not too bad a ratio if a provider is hosting a small libary, but Mangione said it gets exponentially more expensive as providers try to address the long tail of subscriber interest. More content doesn’t just mean adding more servers, but it means adding enough servers to maintain performance levels even in the case of unexpected traffic, thus drastically reducing the number of viewers each server actually supports.

In an email, Andrew Grant, Akamai’s senior manager for entertainment industry marketing, described the problem to me as in this manner:

For VOD content, as content libraries get larger – i.e. As more and more of the ‘long-tail’ of video content is made available for consumers to enjoy – the system must be architected to support a projected average amount of streaming per title, and then over-provisioned to support the unpredictable traffic associated with an outlying event. Think when Michael Jackson died and suddenly traffic for old concert footage suddenly spiked. Building a system to support this level of ‘flash crowd’ traffic would require leaving a large amount of server capacity under-utilized a large part of the time.

By Mangione’s math, a 100,000-movie library would cost about $1.4 billion a year to operate, as opposed t0 about $10,000 for a 10,000-movie library. Akamai’s Grant noted that his company’s CDN (which is one of several that serve Netflix’s streaming service) is built in part to adjust to such spikes by adding capacity as needed. Akamai’s services, of course, aren’t free either.

Mangione realized that cable providers could actually save themselves infrastructure costs and add more content by leveraging the CPU and memory present in their set-top boxes. By creating a P2P network of subscribers’ boxes, providers wouldn’t have to build as much infrastructure because their servers would get hit a lot less. Once subscribers downloaded movies, their set-top boxes and Internet connections would handle most of the subsequent downloads from other subscribers.

But there’s a problem with such a simple approach: Movie files are large, and P2P networks can be slow because traffic might have to traverse multiple network hops. Even if they offer fewer titles, streaming services such as Netflix might look more appealing because their use of CDNs allows them to deliver content so much faster.

Enter big data

The solution to this problem, Mangione decided, is to determine subscribers’ particular interests and then create geographically logical clusters of subscribers who share the same interests. The benefits to this approach are twofold:

1. It decreases the chances of having to hit a provider’s central server because the chances are higher that someone (or multiple people) in a user’s cluster has that title.
2. It improves the accuracy of recommendations because the system primarily analyzes what people with the same interests are viewing and doesn’t have to filter through as much noise from the overall subscriber base.

Such private networks scale to about 10,000 users, Mangione said.

Aside from classifying users into groups based on what they watch, Media Moguls captain and Opera principal scientist Frank Elliott noted the system also could determine the genre of movies more accurately based on who’s watching them. Ideally, the algorithms would maximize P2P traffic and minimize server load by accurately pointing subscribers to new content they want but don’t have, and that’s on other set-top boxes in their clusters.

Opera does have some clout when it comes to talking about recommendation algorithms: A team that included several Opera employees came in second place in the Netflix Prize competition in 2009, although it and the winning team finished in a statistical tie.

Is it viable?

It is, of course, up to cable or satellite providers to determine the viability of Opera’s model and implement it, but the signs point to that happening. For one, Opera CEO Arnab Gupta said his company’s plan is to commercialize the technology through a media-industry partner that already has the data, infrastructure and distribution plans in place. Ideally, that would be a cable or satellite provider (Dish Network is ramping up its streaming service), but conceivably could be an Internet-only device that has adequate hardware specs.

And although set-top boxes might be relics of the past as new boxless viewing methods come into existence, mainstream ubiquity could be a way off. Adopting the Opera method would let providers leverage their existing networks of set-top boxes in the meantime, and its relevancy might well carry into future viewing methods. Whether customers are streaming content from other devices (e.g., game consoles, Roku, etc.) or straight through their TVs, providers still have to worry about the backend costs of storing and delivering all that content.

In any case, licensing content is also a hurdle, but all the hubbub about the Stop Online Piracy and Protect-IP Acts might end up proving to be a blessing in disguise. If the public outcry against these bills is enough to kill them in the House and Senate, it might well serve as a signal to content owners to focus on new, legal distribution channels rather than on lawsuits. Something like this, done through existing cable or satellite partners, could be one good option.

Feature image courtesy of Flickr user PascalSijen; other images courtest of Opera Solutions.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• 9 Companies that Pushed the Infrastructure Discussion in 2010

Opera’s story

Opera has been flying under the radar, but it hasn’t exactly been struggling. Opera CEO Arnab Gupta told me in May that the company was already doing $100 million a year in revenue, growing fast since launching in 2004. It was able to grow so fast because of the inherent value of big data analytics and the Opera service. Gupta told me in a call this morning that he expects the company to do about $140 million next year.

According to the Wall Street Journal, the funding values the company at about $500 million.

Opera does such good business in part because large customers such as Citicorp and Nissan don’t need much convincing that there’s value in gleaning intelligence from their data. But unlike many other big data products, Opera takes the guesswork out of the process. It helps customers determine the best strategies, and then processes and analyzes the data as a service. Customers don’t have to acquire analytics expertise, nor do they have to invest in expensive infrastructure.

It’s one of a handful of companies doing big data in the cloud, which is a trend I think will catch on in a major way over the next couple of years.

Opera also has put a focus on employing Ph.Ds. — currently about 90, with several dozen more having advanced scientific degrees — something that makes it more akin to blue-chipper IBM than to many of today’s big data startups pushing Hadoop or NoSQL technologies (although most, such as Cloudera, have at least a few Ph.Ds. on board). That might have something to do with the nature of predictive analytics and machine intelligence, which are very focused on mathematics and algorithms to continually improve accuracy.

Gupta also explained that Opera employs a large number of subject-matter experts in areas such as fraud detection, retail and marketing, which helps the company create relevant strategies based on a customer’s business. As I wrote in May:

“Signals” are the key to how Opera does what it does. … Gupta explains signals as pieces of information that have value in any given field. Opera, Gupta says, is built to “mine the signal versus mine the data [itself].”

Now’s the time to raise big data money

Opera sought funding, Gupta told me, because the company had pretty much reached its limits in terms of growing the business based on word of mouth. To grow to where it wants to be — somewhere between a niche player and IBM — he said Opera needs to step up its sales efforts and marketing push. Based on discussions with CIOs, Gupta thinks the analytics market will crest around the end of next year as CIOs face enormous data spikes, and Opera wants to have the funds to capitalize on the demand.

He also thinks some acquisition are in order, particularly around web-scraping technologies that help find pockets of relevant data wherever they’re hiding on the web. Otherwise, Gupta said, Opera will look to acquire small companies started by eccentric scientists without great business acumen, just to bring their minds on board, and niche companies focused on specific vertical markets.

If Opera’s goal really is to remain independent, its size and valuation might help it do that. At 600 employees and valued at nearly half a billion dollars, buying and integrating the company presumably would be a bigger deal for an Oracle, HP, IBM or Dell wanting to bolster its big data business than would be snatching up a young pure-technology startup and building a service play around the software.

One interesting nugget about Opera’s funding is that it got more than $50 million than it initially wanted, but it had to work to get it. The company is based in New Jersey, but Gupta said East Coast investors had a hard time buying in, partially because they were focused on analyzing Opera’s revenue numbers instead of looking at the technology and the big data space. In Silicon Valley, he said, investors had a far easier time seeing his company’s potential.

Feature image courtesy of Flickr user dullhunk.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data

Thankfully, big data and the cloud have already found each other. Although it’s still very early in the evolution of this combination — experts predict major investment in this area going forward — several companies have already melded the two into a variety of unique services.

Quantivo. Quantivo just announced its foray into the space Tuesday with a cloud-based version of its analytics platform. The platform combines business data from multiple sources, transforms and enriches it, then lets customers work with it via Quantivo’s specialized interface. Because knowing the right questions to ask is often one of the more difficult aspects of big data, Quantivo says its technology also takes some of the guesswork out of the process by “intelligently auto-compiling lists of patterns” in customers’ datasets.

1010data. 1010data actually has been doing big data as a service for more than a decade, before anyone was talking about the cloud. It provides a variety of services for specific big data use cases, including data warehousing, business intelligence advanced analytics. Although 1010data utilizes Hadoop and other traditional big data tools to power its service, its Customers interact with the service using familiar tools such as spreadsheets that make it easier to find the connections and trends they’re looking for.

Opera Solutions. Opera Solutions is an interesting company, because although it’s doing $100 million in revenue a year, relatively few people have heard of it. But its service is pretty compelling: Customers upload their data to Opera’s platform, which then analyzes it and delivers results based on the relevant “signals” in a customer’s data set. Not content with providing generic analysis to customers, Opera focuses on each customer’s specific needs and employs experts in a variety of industries to help it cater unique analytics programs for each customer.

IBM. IBM has seemingly limitless options in terms of providing big data analytics as a cloud-based service, but its current strategy appears centered around Hadoop. When IBM launched its SmartCloud cloud computing platform in April, it promised that Hadoop workloads will be part of it. A likely candidate to provide that capability is InfoSphere BigInsights, IBM’s Hadoop-based software for analyzing and visualizing large quantities of unstructured data. BigInsights previously was available as a service on IBM’s test-and-development cloud that SmartCloud replaced.

Amazon Web Services. AWS isn’t providing actual analytics as a service, just the parallel processing framework and computing power necessary to do them at scale. Its Elastic MapReduce platform is a cloud-based Hadoop implementation onto which users port their Hadoop applications, then upload their data and run the workload. Like all things AWS, customers only pay for the resources used while the job’s running, as well as for storing the data in AWS’s S3 storage service.

HPCC Systems. LexisNexis spinoff and Hadoop alternative HPCC Systems plans to give customers cloud-based access to a system running the company’s HPCC data-processing software. During an interview during Structure 2011, CTO Armando Escalante noted the company might even offer up its own massive data sets — which span the financial, legal and intelligence sectors, among others — to be processed by customers’ applications.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Takeaways from the second quarter in cloud and data
• Cloud computing infrastructure: 2012 and beyond

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• Infrastructure Overview, Q2 2010