Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

The new application, called The Wall, is essentially a way to make the customer service experience more palatable for consumers and to lower the burden of hiring new representatives. Because it’s designed to look and function like Facebook, MetLife CIO and SVP of Regional Application Development Gary Hoberman told me, The Wall means new hires don’t have to be trained on complex enterprise call center software. For customers calling MetLife to discuss a claim or their coverage, it means fewer annoying waits as an agent accesses data from any of dozens of different places.

“Instead of seeing what someone had for dinner, [The Wall is] all a customer’s transactions,” Hoberman said. Claims, records, status, possible cross-sell information (e.g., if someone lives in an apartment and might need renter’s insurance) — it’s all in there. Looking forward, he said, it might even contain other publicly available information from social media and certain mobile apps that would give the company even greater visibility into its customers’ lives.

Up and running in 3 months, on MongoDB

From a business perspective, though, the most-impressive part of The Wall is how quickly it was implemented and what a divergence from classic large-enterprise IT practices it represents. For Hoberman, who spent 16 years at Citi before joining MetLife in mid-2012, the process was eye-opening. If you told someone in the financial services industry that it would take just five days to get servers up and running for the prototype of such a big application, he said, “they’d look at you like you had two heads.”

But that’s exactly what MetLife did. In fact, it had the entire prototype built just two weeks after devising it and the production system up and running in just three months. It came together so fast because of MetLife’s new focus on cutting-edge IT and clear mission to build a useful product rather than, as Hoberman put it, “doing big data for big data’s sake.” The tech team was willingly working nights and weekends and the leadership team was directly involved because everyone understood what a fundamental change the application could have on the business.

“In insurance,” Hoberman said, “… working in months, not years, is really a startup mentality.”

How big an undertaking was it? Built atop MongoDB, The Wall brings together data from more than 70 legacy systems and merges it into a single record. It runs across six servers in two data centers and presently stores about 24 terabytes of data. That includes MetLife’s entire U.S. customer base (some 45 million agreements in total), although the goal is to expand it to international customers and multiple languages, as well, and maybe even create a customer-facing version. It updates in near real time, just like the Facebook wall, as new customer data is entered.

Building a production database system on NoSQL technology isn’t commonplace in insurance or other large industries, but it was about the only way to pull this off. Going with the relational model, Hoberman explained, would have meant figuring out a common set of schema across such a wide range of products (insurance products and terms vary from state to state and country to country) that it would have been nearly impossible to actually achieve that coveted 360-degree customer view. MongoDB let Hoberman’s team build some light schema to give the app order, but to be able to take in all the data it had available.

Bringing in new tech, and new blood

This is only a part of what MetLife is doing with new information technologies, though, and only a fraction of what it wants to do. With The Wall, specifically, MetLife Hoberman wants to build next-best action models that will give agents guidance on how to best deal with customers. Elsewhere, the company has already used its new centralized MongoDB system to build models for predicting attrition, and it’s using Hadoop and HBase for some other workloads where they’re a better fit.

It’s all thanks to a company mandate to save $450 million from its bloated technology and operations budget and then invest two-thirds of it back into new technology. “We literally have a $300 million investment to decide what’s going to be the future of MetLife,” Hoberman said. It’s kind of like being in a startup, he added, only with the resources to make sure everything is done right (much like with other large enterprises embracing open source, Hoberman’s team prototyped The Wall using open source MongoDB but brought in 10gen when it came time to build a production system).

It might be easy to mock that statement, except that Hoberman and his peers are putting their money where their mouths are by bringing in new talent, as well. It’s setting up a team in the Research Triangle region of North Carolina and bringing in employees with expertise in areas such as social, mobile and big data. And Hoberman is far less concerned with specific technical skills than he is with motivation.

It’s all about “attitude and aptitude,” he said. “They can learn anything.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How search can unlock the power of big data
• Takeaways from the second quarter in cloud and data

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Big data 2013: key trends and companies to watch

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

Results for linear regression test

AMPLab’s most well-known product in the big data space, called Spark, is an in-memory parallel processing framework that’s comparable to Hadoop MapReduce except, its creators claim, it is up to 100 times faster. Because it runs in-memory, Spark might be comparable with something like Druid or SAP’s HANA system, too. Spark is the processing engine that powers ClearStory’s next-generation analytics and visualization service.

Like Hive as a data warehouse for Hadoop? Then you’ll love Shark, which is short for “Hive on Spark.”

Even as Hadoop gets more flexible thanks to new features such as YARN, which would technically allow it to run an alternative framework like Spark, AMPLab has its own cluster-management project called Mesos. Twitter is a big fan of Mesos, which is also an Apache Incubator project.

AMPLab’s latest target is the Hadoop Distributed File System, or HDFS. HDFS has long been criticized for availability and speed, so AMPLab created an alternative called Tachyon (hat tip to High Scalability for calling my attention to it). According to the Tachyon homepage, “it offers up to 300 times higher throughput than HDFS, by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, and enables different jobs/queries and frameworks to access cached files at memory speed.”

AMPLab isn’t the first to question the cult of HDFS, though. There are numerous commercial options available, and Quantcast built its own open source file system that it claims is faster and more efficient when running at massive scale.

But it’s probably unfair to call AMPLab’s projects competitors to Hadoop. They’re certainly alternatives, but they’re also complementary, as Twitter’s heavy use of Hadoop and Mesos demonstrates. And Spark, Shark, Mesos and Tachyon are all compatible with their peer projects from the Apache Hadoop project.

Really, AMPLab is doing what any research institution does by pushing the limits of the current commercially available software. If it happens to disrupt the status quo, then so be it. For users, though, it’s just providing some new options to play around with as they try to find the right tool for their particular jobs. Its partners and sponsors, including Google, Facebook, Microsoft and Amazon Web Services, certainly have an interest in finding the best-possible technology, or creating it if necessary.

The MLBase architecture.

Other related AMPLab projects include PIQL, a SQL-like query language that sits atop a key-value store; MLBase, a system for doing machine learning on distributed systems; Akaros, an operating system for manycore and large SMP systems; and Sparrow, a cluster-scheduling system designed for low-latency computing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

Last fall, Stanford launched its own open-source online learning platform Class2Go, which it released to the public in January. Developed by a team of Stanford engineers, the platform was designed to support the university’s online classes and research. In addition to being open, the platform was intended to be inter-operable with other services and portable (meaning that the course content isn’t tied to one platform). But as part of the new collaboration, Stanford will cease development on that platform and focus its efforts on edX.

“[We'll] fold in the key features of the Class2Go platform in the open-source edX and, together, we’ll be working on a single platform going ahead,” Anant Agarwal, president of edX, said on a call with reporters. “By putting all the wood behind one arrow, so to speak, we thought we could have a bigger impact.”

Since its launch, other schools around the world have started using Class2Go. While the platform will continue to be available to other users, John Mitchell, Stanford’s vice provost for online learning, said they’ll work with those schools to migrate to edX while it transitions its own courses.

The two organizations gave few details on how the collaboration would actually work. But they said that Class2Go’s analytics tools, which can track how long students watch a given video, which sections they repeat and other kinds of student activity on the site, are an example of the kinds of features that will be integrated with edX.

Despite Stanford’s collaboration on the edX platform, Mitchell said the university was not joining the “X University Consortium” of institutions that offer courses on the edX site — which is not entirely surprising given its affiliation with for-profit rival Coursera. The startup was launched by two Stanford professors and the university was one of its launch partners.

But even as Stanford and other top universities partner with for-profit online course providers, like Coursera and Udacity, the growing support for an open source platform shows that schools want to experiment with multiple approaches and be able to control and customize online educational courses and learning tools. The open-source approach means developers anywhere can add new tools to the platform, that professors can create online experiences that best suit their needs and that schools can learn from the innovation of others.

In addition to the Stanford partnership, edX also announced that on June 1, it will release the entire source code for the online learning platform. That development follows its announcement last month that it would release its XBlock SDK, the underlying architecture supporting edX course content.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• What the Google-Motorola deal means for Android, Microsoft and the mobile industry
• Open-sourcing the food industry: new technology for a new food system

Although, of all LinkBench’s features — and you can read all about them in a Facebook Engineer wall post from Monday morning — probably the biggest is that it’s open source and built to be extensible. One of the biggest problems with benchmarks overall is that they rarely align with actual production workloads inside the companies that are supposed to care about them. In this case, for example, a benchmark for measuring the performance of Facebook’s massive MySQL+memcached+Flashcache database architecture against its massive social graph and transaction activity would be all but worthless unless someone was just planning to rebuild Facebook.

I’ve written in the past that perhaps crowdsourced benchmarks are the wave of the future: essentially a compiled set of statistics and best practices as more companies test different database (or Hadoop) technologies on different hardware setups against different workloads and publish the results. Everything will of course vary by the exact details within any given environment, but it would be a good way to get a sense of how a particular stack might, or perhaps should, fare.

But an open source benchmark tuned for a specific use case — social graphs — by probably the world’s foremost expert on that use case is interesting, too. Anyone else trying to serve data from their own social graphs can benefit from some of LinkBench’s more-prominent features, such as its ability to generate “large synthetic social graphs,” while tuning it to the specifics of their own infrastructure. After all, it might be that your app has different requirement around reading versus writing data, and it’s very possible you’re not using MySQL, either.

Or maybe you are using MySQL and want to see how a newer database technology might handle your graph workload. That, by the way, is one of the reasons Facebook built LinkBench, according to this post.

At any rate, the social web is all about graphs, and database performance really matters for anyone trying to build a service that stays online and provides a pleasant user experience. Say what you want about Facebook, but its services perform, so the bar is set high for anyone trying to dethrone it or at least to build something than can attract an equally large and devout following.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Is the Future of Enterprise Completely Open Source?
• How devops can reduce cycle times
• Migrating media applications to the private cloud: best practices for businesses

MapR’s decision to open source parts of it code is significant, but not groundbreaking. The company is only releasing its improvements to a handful of Hadoop-related Apache projects that are included in the MapR distribution of Hadoop, but not the proprietary code that’s MapR’s real competitive advantage in the contentious Hadoop market. While it’s still not flying the all-open-source banner like Hortonworks is, the code release puts MapR more on par with competitor Cloudera, which bolsters its open source aspects with some proprietary software for managing Hadoop clusters.

MapR also took another step in the open source direction on Thursday, announcing a partnership with Canonical that integrates MapR’s M3 distribution with the Ubuntu Linux operating system. The two also have plans to ease the installation of MapR’s Hadoop software on OpenStack-based cloud infrastructure.

I wrote recently in relation to MapR’s $30 million VC investment that the company is in a tricky position when it comes to open source. The Hadoop ecosystem was built on open source and still values it immensely, but some customers are definitely willing to pay money for products that deliver the features they want, open source or not.

As for Yelp, well, it’s just following in the footsteps of many companies — Netflix and everyone doing something on Kaggle (including GigaOM) — in trying to find new ways to use its data. The data set it’s releasing is from the Phoenix, Ariz., area and include 11,537 businesses, 8,282 checkin sets, 43,873 users and 229,907 reviews. The deadline for entries is May 20, and they can be submitted in pretty much any form you can imagine.

Hopefully, for Yelp’s sake, it doesn’t step in it the way other companies — including Netflix and AOL — have when they released supposedly anonymous data sets that were later de-anonymized. Releasing data sets gives clear benefits to both the source companies and institutions or individuals accessing the data, but privacy snafus have a away sneaking up and mitigating some of the goodwill.

Feature image courtesy of Shutterstock user Jakub Krechowicz.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How search can unlock the power of big data
• Infrastructure Q1: Cloud and big data woo enterprises

In practice, Cascading is generally used as a higher-level method than MapReduce for writing Hadoop jobs, although it’s technically a framework that could support any number of distributed-processing frameworks. It’s used by a number of notable users, including Etsy, Airbnb and Climate Corporation. In February, the Cascading project expanded its scope to address the growing SQL-on-Hadoop trend with a project called Lingual.

Software veteran Gary Nakamura is taking on the role of Concurrent CEO, replacing Cascading creator Chris Wensel, who’ll stay on as the company’s CTO.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

At least until now. As of Wednesday, Riak CS will be available under the Apache 2 license.

This is part of an evolution, said Basho CTO Justin Sheehy (pictured above.) Last fall, Basho said it would work with Citrix to integrate Apache Cloudstack effort to integrate Riak CS into Cloudstack.

“We are not part of the OpenStack Foundation but we have deployed Riak CS in OpenStack environments with OpenStack components,” Sheehy said. OpenStack also includes its own storage subsystem as an option.  The work with Cloudstack is a little more explicit since Cloudstack is mostly a compute cloud with networking elements but does not itself offer a real storage subsystem analogous to Amazon’s S3 storage, he said.

One of Riak CS’s key draws is that customers can use it to build private or public cloud storage that is API-compatible with Amazon S3, the 800-lb gorilla in public cloud storage.

Along with its licensing news, Basho also unveiled new features for the open source version including multipart uploads to ease the movement of very large files in pieces into the store; per-bucket policies to restrict access to some data based on its source IP; and Riak Control, a standalone web administration tool for managing users (see screenshot below.)

Basho continues to offer Riak CS Enterprise as a commercial product with around-the-clock support.  To be clear, Basho’s Riak, the distributed database underlying Riak CS was already an open source product.

Sheehy will be talking big data and other hot topics at GigaOM’s Structure: Data event in New York on Wednesday.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• What Enterprise Software Vendors Could Learn from the Consumer Space
• How search can unlock the power of big data