Kidding aside, that’s actually quite an ambitious goal in a Hadoop market that’s big and growing, but that’s exemplified by expensive consulting arrangements and purpose-built applications. Even more so for companies that want to do something other than transforming unstructured data into structured data (often called ETL) or run back-office analytics jobs. In fact, WibiData has spent the last 18 months doing just this type of deal, and Bisciglia says every single customer has already engaged with one of the big three Hadoop vendors (Cloudera, Hortonworks and MapR).

Home energy-management startup Opower is a good example of this process. It’s actually one of Cloudera’s banner customers, but “when they wanted to take [their software-as-a-service tool] beyond batch analysis and ETL workloads,” Bisciglia said, Opower came to WibiData. So whereas the Opower service was originally focused on nightly data analysis comparing users’ energy usage against that of other users, it’s now working on dynamic recommendations for users and letting them engage with the application in new ways.

The WibiData architecture

During these engagements, WibiData has been building up its core technology for connecting those brawny back-office Hadoop environments to predictive customer-facing applications – a collection of HBase, data-formatting tools and machine learning algorithms that the company has been slowly open-sourcing under the Kiji banner. It has also been learning the similarities among the applications it’s building for customers in the same field, figuring out what’s repeatable. What does any given company in the retail space, for example, need to get started on its own recommendation engine?

And now, Bisciglia says, WibiData is going to double down on building application software based on what it has learned. The first two industries it targets will likely be financial services and retail, two areas where the company has seen a lot of traction. He envisions the finished product including some pre-defined schema for formatting data and some pre-built predictive models, both broadly applicable across that industry rather than specific to a single user.

There will also be different interfaces that allow different types of users (e.g., data scientists, systems engineers and business users) to interact with the data in the ways they need to.

Time will tell if WibiData can actually accomplish its goal of turning Hadoop into a collection of somewhat specialized software packages, but someone has to. Even industry heavyweights like Cloudera see the need, but their hands are full just getting Hadoop integrated into existing environments and getting those early uses up and running. As Cloudera CEO Mike Olson said at Structure: Data in 2012 to anyone ambitious enough to tackle the Hadoop-application gap, “Call me, I’ll connect you with funding. The money is out there.”

If you want to hear more about the need for Hadoop applications, check out this panel from Structure: Data 2013, where I speak with WibiData’s Omer Trajman, Continuuity’s Jonathan Gray and Pivotal’s Muddu Sudhakar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Why the big data startup boom will likely be short-lived

The reason for creating Pattern is pretty simple, according to Concurrent founder and CTO Chris Wensel: “Hadoop is never used alone.” It’s always part of a data environment that also includes databases, visualization tools, analytics software and/or statistical analysis tools that arguably do the really valuable work. Hadoop’s real value is an integration platform that can feed data into these other systems and, ideally, put their outputs to work across much larger datasets.

Developers can use the Pattern Java API to create machine learning jobs, but they can also simply export a Predictive Model Markup Language (PMML) file from software like R, SAS and MicroStrategy that Pattern will read and run them as a Cascading workflow. Models are useless unless you can run them in production, Wensel said, and Pattern lets them run across more data, stored in Hadoop, than you can use to build them with those other tools.

However, Wensel noted, “The real takeaway isn’t Pattern itself.”

From his perspective, the real story is Pattern plus Cascading plus Lingual, the open source SQL-to-Hadoop tool that Concurrent recently developed and released. Lingual is the tie that binds everything together, creating a sort of assembly line for data as it works its way from generation to delivering some value. For example, someone might create a Cascading job that adds structure to incoming data, and then pull some of the data into R using Lingual. Once a model is created in R and exported to the Hadoop cluster using Pattern, Lingual can feed the MapReduce output file back to R so a data scientist can test the model’s accuracy.

And actually, Wensel said, Lingual could have a positive effect on companies’ bottom lines. Airbnb recently replaced a departed engineer with Lingual for monthly migrations of data from Hadoop and into SQL environments. Climate Corporation, a massive Hadoop and Cascading user, could use Lingual to let its crop-and-weather insurance customers access their data from the company’s Hadoop store.

Lingual and Pattern should help Concurrent finally make some money, too. Both of them, as well as the Cascading framework that underpins them, will always be open source, Wensel said, but it plans to create “a suite of products that will make your life much better if … you standardize on Cascading.”

For example, the company has the ability to monitor jobs at the application level rather than the cluster level, meaning it can tell you the details of that job that’s locking up all the resources and whether you really want to kill it (it might be an important report for the CFO …). “We can do some really interesting things,” Wensel said.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

This post was updated at 2:48pm PT to correct Chris Wensel’s title. He is CTO.

Feature image courtesy of Shutterstock user PENGYOU91.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

The new application, called The Wall, is essentially a way to make the customer service experience more palatable for consumers and to lower the burden of hiring new representatives. Because it’s designed to look and function like Facebook, MetLife CIO and SVP of Regional Application Development Gary Hoberman told me, The Wall means new hires don’t have to be trained on complex enterprise call center software. For customers calling MetLife to discuss a claim or their coverage, it means fewer annoying waits as an agent accesses data from any of dozens of different places.

“Instead of seeing what someone had for dinner, [The Wall is] all a customer’s transactions,” Hoberman said. Claims, records, status, possible cross-sell information (e.g., if someone lives in an apartment and might need renter’s insurance) — it’s all in there. Looking forward, he said, it might even contain other publicly available information from social media and certain mobile apps that would give the company even greater visibility into its customers’ lives.

Up and running in 3 months, on MongoDB

From a business perspective, though, the most-impressive part of The Wall is how quickly it was implemented and what a divergence from classic large-enterprise IT practices it represents. For Hoberman, who spent 16 years at Citi before joining MetLife in mid-2012, the process was eye-opening. If you told someone in the financial services industry that it would take just five days to get servers up and running for the prototype of such a big application, he said, “they’d look at you like you had two heads.”

But that’s exactly what MetLife did. In fact, it had the entire prototype built just two weeks after devising it and the production system up and running in just three months. It came together so fast because of MetLife’s new focus on cutting-edge IT and clear mission to build a useful product rather than, as Hoberman put it, “doing big data for big data’s sake.” The tech team was willingly working nights and weekends and the leadership team was directly involved because everyone understood what a fundamental change the application could have on the business.

“In insurance,” Hoberman said, “… working in months, not years, is really a startup mentality.”

How big an undertaking was it? Built atop MongoDB, The Wall brings together data from more than 70 legacy systems and merges it into a single record. It runs across six servers in two data centers and presently stores about 24 terabytes of data. That includes MetLife’s entire U.S. customer base (some 45 million agreements in total), although the goal is to expand it to international customers and multiple languages, as well, and maybe even create a customer-facing version. It updates in near real time, just like the Facebook wall, as new customer data is entered.

Building a production database system on NoSQL technology isn’t commonplace in insurance or other large industries, but it was about the only way to pull this off. Going with the relational model, Hoberman explained, would have meant figuring out a common set of schema across such a wide range of products (insurance products and terms vary from state to state and country to country) that it would have been nearly impossible to actually achieve that coveted 360-degree customer view. MongoDB let Hoberman’s team build some light schema to give the app order, but to be able to take in all the data it had available.

Bringing in new tech, and new blood

This is only a part of what MetLife is doing with new information technologies, though, and only a fraction of what it wants to do. With The Wall, specifically, MetLife Hoberman wants to build next-best action models that will give agents guidance on how to best deal with customers. Elsewhere, the company has already used its new centralized MongoDB system to build models for predicting attrition, and it’s using Hadoop and HBase for some other workloads where they’re a better fit.

It’s all thanks to a company mandate to save $450 million from its bloated technology and operations budget and then invest two-thirds of it back into new technology. “We literally have a $300 million investment to decide what’s going to be the future of MetLife,” Hoberman said. It’s kind of like being in a startup, he added, only with the resources to make sure everything is done right (much like with other large enterprises embracing open source, Hoberman’s team prototyped The Wall using open source MongoDB but brought in 10gen when it came time to build a production system).

It might be easy to mock that statement, except that Hoberman and his peers are putting their money where their mouths are by bringing in new talent, as well. It’s setting up a team in the Research Triangle region of North Carolina and bringing in employees with expertise in areas such as social, mobile and big data. And Hoberman is far less concerned with specific technical skills than he is with motivation.

It’s all about “attitude and aptitude,” he said. “They can learn anything.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How search can unlock the power of big data
• Takeaways from the second quarter in cloud and data

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Big data 2013: key trends and companies to watch

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

Results for linear regression test

AMPLab’s most well-known product in the big data space, called Spark, is an in-memory parallel processing framework that’s comparable to Hadoop MapReduce except, its creators claim, it is up to 100 times faster. Because it runs in-memory, Spark might be comparable with something like Druid or SAP’s HANA system, too. Spark is the processing engine that powers ClearStory’s next-generation analytics and visualization service.

Like Hive as a data warehouse for Hadoop? Then you’ll love Shark, which is short for “Hive on Spark.”

Even as Hadoop gets more flexible thanks to new features such as YARN, which would technically allow it to run an alternative framework like Spark, AMPLab has its own cluster-management project called Mesos. Twitter is a big fan of Mesos, which is also an Apache Incubator project.

AMPLab’s latest target is the Hadoop Distributed File System, or HDFS. HDFS has long been criticized for availability and speed, so AMPLab created an alternative called Tachyon (hat tip to High Scalability for calling my attention to it). According to the Tachyon homepage, “it offers up to 300 times higher throughput than HDFS, by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, and enables different jobs/queries and frameworks to access cached files at memory speed.”

AMPLab isn’t the first to question the cult of HDFS, though. There are numerous commercial options available, and Quantcast built its own open source file system that it claims is faster and more efficient when running at massive scale.

But it’s probably unfair to call AMPLab’s projects competitors to Hadoop. They’re certainly alternatives, but they’re also complementary, as Twitter’s heavy use of Hadoop and Mesos demonstrates. And Spark, Shark, Mesos and Tachyon are all compatible with their peer projects from the Apache Hadoop project.

Really, AMPLab is doing what any research institution does by pushing the limits of the current commercially available software. If it happens to disrupt the status quo, then so be it. For users, though, it’s just providing some new options to play around with as they try to find the right tool for their particular jobs. Its partners and sponsors, including Google, Facebook, Microsoft and Amazon Web Services, certainly have an interest in finding the best-possible technology, or creating it if necessary.

The MLBase architecture.

Other related AMPLab projects include PIQL, a SQL-like query language that sits atop a key-value store; MLBase, a system for doing machine learning on distributed systems; Akaros, an operating system for manycore and large SMP systems; and Sparrow, a cluster-scheduling system designed for low-latency computing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q2: Big data and PaaS gain more momentum

Last fall, Stanford launched its own open-source online learning platform Class2Go, which it released to the public in January. Developed by a team of Stanford engineers, the platform was designed to support the university’s online classes and research. In addition to being open, the platform was intended to be inter-operable with other services and portable (meaning that the course content isn’t tied to one platform). But as part of the new collaboration, Stanford will cease development on that platform and focus its efforts on edX.

“[We'll] fold in the key features of the Class2Go platform in the open-source edX and, together, we’ll be working on a single platform going ahead,” Anant Agarwal, president of edX, said on a call with reporters. “By putting all the wood behind one arrow, so to speak, we thought we could have a bigger impact.”

Since its launch, other schools around the world have started using Class2Go. While the platform will continue to be available to other users, John Mitchell, Stanford’s vice provost for online learning, said they’ll work with those schools to migrate to edX while it transitions its own courses.

The two organizations gave few details on how the collaboration would actually work. But they said that Class2Go’s analytics tools, which can track how long students watch a given video, which sections they repeat and other kinds of student activity on the site, are an example of the kinds of features that will be integrated with edX.

Despite Stanford’s collaboration on the edX platform, Mitchell said the university was not joining the “X University Consortium” of institutions that offer courses on the edX site — which is not entirely surprising given its affiliation with for-profit rival Coursera. The startup was launched by two Stanford professors and the university was one of its launch partners.

But even as Stanford and other top universities partner with for-profit online course providers, like Coursera and Udacity, the growing support for an open source platform shows that schools want to experiment with multiple approaches and be able to control and customize online educational courses and learning tools. The open-source approach means developers anywhere can add new tools to the platform, that professors can create online experiences that best suit their needs and that schools can learn from the innovation of others.

In addition to the Stanford partnership, edX also announced that on June 1, it will release the entire source code for the online learning platform. That development follows its announcement last month that it would release its XBlock SDK, the underlying architecture supporting edX course content.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• What the Google-Motorola deal means for Android, Microsoft and the mobile industry
• Open-sourcing the food industry: new technology for a new food system

Although, of all LinkBench’s features — and you can read all about them in a Facebook Engineer wall post from Monday morning — probably the biggest is that it’s open source and built to be extensible. One of the biggest problems with benchmarks overall is that they rarely align with actual production workloads inside the companies that are supposed to care about them. In this case, for example, a benchmark for measuring the performance of Facebook’s massive MySQL+memcached+Flashcache database architecture against its massive social graph and transaction activity would be all but worthless unless someone was just planning to rebuild Facebook.

I’ve written in the past that perhaps crowdsourced benchmarks are the wave of the future: essentially a compiled set of statistics and best practices as more companies test different database (or Hadoop) technologies on different hardware setups against different workloads and publish the results. Everything will of course vary by the exact details within any given environment, but it would be a good way to get a sense of how a particular stack might, or perhaps should, fare.

But an open source benchmark tuned for a specific use case — social graphs — by probably the world’s foremost expert on that use case is interesting, too. Anyone else trying to serve data from their own social graphs can benefit from some of LinkBench’s more-prominent features, such as its ability to generate “large synthetic social graphs,” while tuning it to the specifics of their own infrastructure. After all, it might be that your app has different requirement around reading versus writing data, and it’s very possible you’re not using MySQL, either.

Or maybe you are using MySQL and want to see how a newer database technology might handle your graph workload. That, by the way, is one of the reasons Facebook built LinkBench, according to this post.

At any rate, the social web is all about graphs, and database performance really matters for anyone trying to build a service that stays online and provides a pleasant user experience. Say what you want about Facebook, but its services perform, so the bar is set high for anyone trying to dethrone it or at least to build something than can attract an equally large and devout following.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Is the Future of Enterprise Completely Open Source?
• How devops can reduce cycle times
• Migrating media applications to the private cloud: best practices for businesses