Kidding aside, that’s actually quite an ambitious goal in a Hadoop market that’s big and growing, but that’s exemplified by expensive consulting arrangements and purpose-built applications. Even more so for companies that want to do something other than transforming unstructured data into structured data (often called ETL) or run back-office analytics jobs. In fact, WibiData has spent the last 18 months doing just this type of deal, and Bisciglia says every single customer has already engaged with one of the big three Hadoop vendors (Cloudera, Hortonworks and MapR).

Home energy-management startup Opower is a good example of this process. It’s actually one of Cloudera’s banner customers, but “when they wanted to take [their software-as-a-service tool] beyond batch analysis and ETL workloads,” Bisciglia said, Opower came to WibiData. So whereas the Opower service was originally focused on nightly data analysis comparing users’ energy usage against that of other users, it’s now working on dynamic recommendations for users and letting them engage with the application in new ways.

The WibiData architecture

During these engagements, WibiData has been building up its core technology for connecting those brawny back-office Hadoop environments to predictive customer-facing applications – a collection of HBase, data-formatting tools and machine learning algorithms that the company has been slowly open-sourcing under the Kiji banner. It has also been learning the similarities among the applications it’s building for customers in the same field, figuring out what’s repeatable. What does any given company in the retail space, for example, need to get started on its own recommendation engine?

And now, Bisciglia says, WibiData is going to double down on building application software based on what it has learned. The first two industries it targets will likely be financial services and retail, two areas where the company has seen a lot of traction. He envisions the finished product including some pre-defined schema for formatting data and some pre-built predictive models, both broadly applicable across that industry rather than specific to a single user.

There will also be different interfaces that allow different types of users (e.g., data scientists, systems engineers and business users) to interact with the data in the ways they need to.

Time will tell if WibiData can actually accomplish its goal of turning Hadoop into a collection of somewhat specialized software packages, but someone has to. Even industry heavyweights like Cloudera see the need, but their hands are full just getting Hadoop integrated into existing environments and getting those early uses up and running. As Cloudera CEO Mike Olson said at Structure: Data in 2012 to anyone ambitious enough to tackle the Hadoop-application gap, “Call me, I’ll connect you with funding. The money is out there.”

If you want to hear more about the need for Hadoop applications, check out this panel from Structure: Data 2013, where I speak with WibiData’s Omer Trajman, Continuuity’s Jonathan Gray and Pivotal’s Muddu Sudhakar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Why the big data startup boom will likely be short-lived

Alone, Hadoop is a software market that IDC predicts will be worth $813 million in 2016 (although that number is likely very low), but it’s also driving a big data market the research firm predicts will hit more than $23 billion by 2016. Since Cloudera launched in 2008, Hadoop has spawned dozens of startups and spurred hundreds of millions in venture capital investment since 2008.

In this four-part series, we’ll explain everything anyone concerned with information technology needs to know about Hadoop. Part I is the history of Hadoop from the people who willed it into existence and took it mainstream. Part II is more graphic; a map of the now-large and complex ecosystem of companies selling Hadoop products. Part III is a look into the future of Hadoop that should serve as an opening salvo for much of the discussion at our Structure: Data conference March 20-21 in New York. Finally, part IV will highlight some the best Hadoop applications and seminal moments in Hadoop history, as reported by GigaOM over the years.

Wanted: A better search engine

Almost everywhere you go online now, Hadoop is there in some capacity. Facebook, eBay, Etsy, Yelp , Twitter, Salesforce.com — you name a popular web site or service, and the chances are it’s using Hadoop to analyze the mountains of data it’s generating about user behavior and even its own operations. Even in the physical world, forward-thinking companies in fields ranging from entertainment to energy management to satellite imagery are using Hadoop to analyze the unique types of data they’re collecting and generating.

Everyone involved with information technology at least knows what it is. Hadoop even serves as the foundation for new-school graph and NoSQL databases, as well as bigger, badder versions of relational databases that have been around for decades.

But it wasn’t always this way, and today’s uses are a long way off from the original vision of what Hadoop could be.

Doug Cutting

When the seeds of Hadoop were first planted in 2002, the world just wanted a better open-source search engine. So then-Internet Archive search director Doug Cutting and University of Washington graduate student Mike Cafarella set out to build it. They called their project Nutch and it was designed with that era’s web in mind.

Looking back on it today, early iterations of Nutch were kind of laughable. About a year into their work on it, Cutting and Cafarella thought things were going pretty well because Nutch was already able to crawl and index hundreds of millions of pages. “At the time, when we started, we were sort of thinking that a web search engine was around a billion pages,” Cutting explained to me, “so we were getting up there.”

There are now about 700 million web sites and, according to Wired’s Kevin Kelly, well over a trillion web pages.

But getting Nutch to work wasn’t easy. It could only run across a handful of machines, and someone had to watch it around the clock to make sure it didn’t fall down.

Mike Cafarella

“I remember working on it for several months, being quite proud of what we had been doing, and then the Google File System paper came out and I realized ‘Oh, that’s a much better way of doing it. We should do it that way,’” reminisced Cafarella. “Then, by the time we had a first working version, the MapReduce paper came out and that seemed like a pretty good idea, too.”

Google released the Google File System paper in October 2003 and the MapReduce paper in December 2004. The latter would prove especially revelatory to the two engineers building Nutch.

“What they spent a lot of time doing was generalizing this into a framework that automated all these steps that we were doing manually,” Cutting explained.

Raymie Stata, founder and CEO of Hadoop startup VertiCloud (and former Yahoo CTO), calls MapReduce “a fantastic kind of abstraction” over the distributed computing methods and algorithms most search companies were already using:

“Everyone had something that pretty much was like MapReduce because we were all solving the same problems. We were trying to handle literally billions of web pages on machines that are probably, if you go back and check, epsilon more powerful than today’s cell phones. … So there was no option but to latch hundreds to thousands of machines together to build the index. So it was out of desperation that MapReduce was invented.”

Parallel processing in MapReduce, from the Google paper

Over the course of a few months, Cutting and Cafarella built up the underlying file systems and processing framework that would become Hadoop (in Java, notably, whereas Google’s MapReduce used C++) and ported Nutch on top of it. Now, instead of having one guy watch a handful of machines all day long, Cutting explained, they could just set it running on between 20 and 40 machines that he and Cafarella were able to scrape together from their employers.

Bringing Hadoop to life (but not in search)

Anyone vaguely familiar with the history of Hadoop can guess what happens next: In 2006, Cutting went to work with Yahoo, which was equally impressed by the Google File System and MapReduce papers and wanted to build open source technologies based on them. They spun out the storage and processing parts of Nutch to form Hadoop (named after Cutting’s son’s stuffed elephant) as an open-source Apache Software Foundation project and the Nutch web crawler remained its own separate project.

“This seem like a perfect fit because I was looking for more people to work on it, and people who had thousands of computers to run it on,” Cutting said.

Cafarella, now an associate professor at the University of Michigan, opted to forgo a career in corporate IT and focus on his education. He’s happy as a professor — and currently working on a Hadoop-complementary project called RecordBreaker — but, he joked, “My dad calls me the Pete Best of the big data world.”

Ironically, though, the 2006-era Hadoop was nowhere near ready to handle production search workloads at webscale — the very task it was created to do. “The thing you gotta remember,” explained Hortonworks Co-founder and CEO Eric Baldeschwieler (who was previously VP of Hadoop software development at Yahoo), “is at the time we started adopting it, the aspiration was definitely to rebuild Yahoo’s web search infrastructure, but Hadoop only really worked on 5 to 20 nodes at that point, and it wasn’t very performant, either.”

Baldeschwieler at Hadoop Summit 2010. Source: Yodel Anectdotal

Stata recalls a “slow march” of horizontal scalability, growing Hadoop’s capabilities from the single digits of nodes into the tens of nodes and ultimately into the thousands. “It was just an ongoing slog … every factor of 2 or 1.5 even was serious engineering work,” he said. But Yahoo was determined to scale Hadoop as far as it needed to go, and it continued investing heavy resources into the project.

It actually took years for Yahoo to moves its web index onto Hadoop, but in the meantime the company made what would be a fortuitous decision to set up what it called a “research grid” for the company’s data scientists, to use today’s parlance. It started with dozens of nodes and ultimately grew to hundreds as they added more and more data and Hadoop’s technology matured. What began life as a proof of concept fast became a whole lot more.

“This very quickly kind of exploded and became our core mission,” Baldeschwieler said, “because what happened is the data scientists not only got interesting research results — what we had anticipated — but they also prototyped new applications and demonstrated that those applications could substantially improve Yahoo’s search relevance or Yahoo’s advertising revenue.”

Shortly thereafter, Yahoo began rolling out Hadoop to power analytics for various production applications. Eventually, Stata explained, Hadoop had proven so effective that Yahoo merged its search and advertising into one unit so that Yahoo’s bread-and-butter sponsored search business could benefit from the new technology.

Cutting (center) flanked by Baldeschwieler and Om Malik at GigaOM’s Hadoop Meetup in 2008.

And that’s exactly what happened, because although data scientists didn’t need things like service-level agreements, business leaders did. So, Stata said, Yahoo implemented some scheduling changes within Hadoop. And although data scientists didn’t need security, Securities and Exchange Commission requirements mandated a certain level of security when Yahoo moved its sponsored search data onto it.

“That drove a certain level of maturity,” Stata said. “… We ran all the money in Yahoo through it, eventually.”

The transformation into Hadoop being “behind every click” (or every batch process, technically) at Yahoo was pretty much complete by 2008, Baldeschwieler said. That meant doing everything from these line-of-business applications to spam filtering to personalized display decisions on the Yahoo front page. By the time Yahoo spun out Hortonworks into a separate, Hadoop-focused software company in 2011, Yahoo’s Hadoop infrastructure consisted of 42,000 nodes and hundreds of petabytes of storage.

From the classroom …

However, although Yahoo was responsible for the vast majority of development during its formative years, Hadoop didn’t exist in a bubble inside Yahoo’s headquarters. It was a full-on Apache project that attracted users and contributors from around the world. Guys like Tom White, a Welshman who actually wrote O’Reilly Media’s book Hadoop: The Definitive Guide despite being what Cutting describes as a guy who just liked software and played with Hadoop at night.

Up in Seattle in 2006, a young Google engineer named Christophe Bisciglia was using his 20 percent time to teach a computer science course at the University of Washington. Google wanted to hire new employees with experience working on webscale data, but its MapReduce code was proprietary, so it bought a rack of servers and used Hadoop as a proxy.

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Real-­time query for Hadoop democratizes access to big data analytics

As WibiData product manager Devjit Chakravarti told me during a recent call, KijiSchema essentially “takes the ‘No’ out of NoSQL.” What he means is that although NoSQL databases such as HBase are lauded in part because they can store unstructured data and don’t require rigid rules for data formatting like relational databases do, having some structure is actually necessary once you want to do meaningful analysis on it. That’s why some commercial products, such as Drawn to Scale’s Spire and Splice Machine’s Splice SQL Engine, already have built functional SQL databases on top of HBase.

Kimball speaking at Structure: Data in 2012
(c) 2012 Pinar Ozger. pinar@pinarozger.com

“If you can’t store data in an organized way, you can’t analyze it effectively,” WibiData Co-Founder and CTO Aaron Kimball explained. KijiSchema isn’t part of WibiData’s secret sauce around predictive analytics for user data, he added, but nothing gets done without it.

Here’s how Kimball describes how KijiSchema manages data in a blog post announcing the project:

“KijiSchema gives developers the ability to easily store both structured and unstructured data within HBase using Avro serialization. It supports a variety of rich schema features, including complex, compound data types, HBase column key and time-series indexing, as well cell-level evolving schemas that dynamically encode version information.

“KijiSchema promotes the use of entity-centric data modeling, where all information about a given entity (user, mobile device, ad, product, etc.), including dimensional and transaction data, is encoded within the same row. This approach is particularly valuable for user-based analytics such as targeting, recommendations, and personalization.”

Kiji resides in the lower left section

The coolest part for HBase developers or prospective HBase developers, however, might be that KijiSchema isn’t just code but is already pre-packaged any ready to deploy. WibiData has created what it calls the Kiji BentoBox — “a fully-functional HBase mini-cluster with KijiSchema on your machine with minimal configuration in under 15 minutes” — that’s available for download on Github.

KijiSchema is also part of a broader Kiji framework for HBase that WibiData plans to open source over the next year or so. People perceive HBase as being complicated to set up and having a steep learning curve, Kimball said, and his teams wants to make it more accessible and lower the barrier for getting started. The ultimate goal is to make the types of HBase applications that folks at Facebook, eBay and other large web shops are building something that any developer can do.

WibiData’s Omer Trajman, formerly VP of technology solutions at Cloudera, describes the ultimate Kiji framework as being akin what the Spring framework if for Java. Despite its complexity, “there are also tens of thousands of developers who have been able to figure [HBase] out,” he said, but learning it might take weeks of intensive training on learning the low-level guts of the Hadoop Distributed File System and other stuff. Why learn to build an enterprise Java application from scratch, Trajman asked, when you can just use Spring?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Cloud and data first-quarter 2013: analysis and outlook

Everything, of course, is relative and we might never see a Hadoop vendor reach the size of a database company such as Oracle with more than 100,000 employees and tens of billions in annual revenue. After all, Hadoop is a new technology for most companies, so it’s not really moving in on an already lucrative market and stealing budgetary dollars from incumbents. Further — and possibly more importantly — the core Hadoop technology is free and open source, meaning there are lots of unpaid downloads so money comes from services, support and large enterprises willing to buy software licenses for value-added products.

Money

Here’s a chart showing how much money Hadoop-based companies have raised thus far (although the grand total will likely rise by at least $10 million next week). Keep in mind, Cloudera only launched in 2009 and Hortonworks launched in June 2011. And these aren’t companies that merely bury Hadoop under an application or can connect their technologies to it — these are companies either selling Hadoop or applications designed specifically for it.

(To view the original, interactive chart, click here.)

In terms of revenue, one might look to a May 2012 report by research from IDC estimating the size of the Hadoop ecosystem to be around $77 million, growing to $813 million by 2016. Those are both impressive numbers, but they might actually be short-changing reality. For one, as I noted at the time, the authors attributed almost no revenue to Amazon Web Services’ Elastic MapReduce service, which is almost certainly generating at least a few million in revenue each year.

Speaking to me in June, Cloudera CEO Mike Olson also took issue with the number, claiming it didn’t even take Cloudera’s revenue into account — which seems entirely possible considering the business Cloudera is doing. I’ve heard from reliable sources that Cloudera is doing very well and is on track to do about $100 million in revenue this year, very possibly more. And as early as April 2011, Cloudera executives were touting that software license revenue had already surpassed services revenue (although it’s arguable whether that will, or even has to, remain the case).

More anecdotally, I’ve heard from several sources that Hortonworks has already declined at least one potentially appealing acquisition offer. That it wouldn’t sell isn’t surprising: sources say the company is valued at $225 million after its last round of funding and is looking to raise more money. And although it just released its first product in June, the company has impressive and potentially lucrative partnerships in place with Microsoft, Teradata, Rackspace, VMware and other large vendors.

MapR, the proprietary thorn in the sides of both Cloudera and Hortonworks, appears to be doing quite well, too. Vice President of Marketing Jack Norris told me in June that his company had higher license revenue than many would expect and predicted that deals with Amazon Web Services and Google Compute Engine would help the company become “the license revenue leader within the next quarter.”

Former Cloudera VP of Technology Solutions Omer Trajan, who just left to join HBase-centric startup WibiData, shared some insights with me from his days at Cloudera that seem to back up vendor confidence. He said most mature production clusters (excluding monster users such as Facebook) consist of about 200 nodes, and many double in size after the first year. That’s part of the reason Cloudera grew in size about 10x during the three years he was there.

“It has definitely been a rocket ship,” he said. “… You just strap in and hope you make it up.”

Interest is only picking up, too: “There are more people that have started big data projects in the past six months than have big big data projects running [in production],” Trajman said.

People

It’s probably not accurate to call companies such as Cloudera, Hortonworks and MapR startups anymore, and we might start to see signs of this shift in personnel moves. Here’s how big they are and expect to become:

• Cloudera: More than 300 employees globally and growing, especially in the sales department.
• Hortonworks: 145 employees as of late October and hiring a person per day, on average, through the end of 2012.
• MapR: More than 125 employees, mostly in technical and engineering positions; starting to build sales team and looks to more than double headcount in 2013.

While Cloudera and Hortonworks, for example, are still young, nimble and agile enough to lure a fair amount of talent from now-officially large enterprises such as VMware, their employees who joined on early and really love the startup life might not stick around.

Trajman’s new home, WibiData, is a fine example of this. It was launched last year by former Cloudera employees Christophe Bisciglia (who actually co-founded Cloudera) and Aaron Kimball to help companies build behavioral-analysis applications on top of Hadoop.

(Maybe there’s a Cloudera mafia shaping up: WibiData’s officemates — MemCachier and Thanx — both count former Cloudera employees as key members or founders of their teams, as does HBase-centric startup Drawn to Scale.)

Trajman, who was one of the first couple dozen employees at Cloudera (and who previously joined Vertica at around the same stage in its growth) told me he likes the rush of getting in the the ground level of new technologies and helping companies do something really new. While he enjoyed establishing and implementing some the the core foundational use cases for Hadoop (e.g., ETL and data exploration) with Cloudera’s early customers, that’s still much of what Cloudera provides to customers because it’s so difficult to build higher-level and higher-value applications at the infrastructural level where Cloudera operates.

“For me, it was very personal in terms of the impact I wanted to have,” Trajman said. At WibiData, he can help users who have the infrastructure part resolved and now want to develop applications that make data analysis a core part of their businesses. Where there’s a focus on innovation, he said, that’s where the innovators go.

This isn’t a bad thing, it’s just a side effect of growth — and when employees stay and innovate in the Hadoop space, it just creates a bigger pie for everyone to share.

Feature image courtesy of Shutterstock user GuskovaNatalia.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Why service providers matter for the future of big data
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Takeaways from the second quarter in cloud and data
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

Details about how, exactly, WibiData goes about letting users do web analytics have been sparse, but co-founder Aaron Kimball, who will present at our Structure: Data conference next month, explained some of it in a blog post on Monday. The post is fairly technical, but the gist is that WibiData leverages Apache Hadoop, HBase and Avro, as well as ample proprietary code, to enable both real-time and batch processing of user data. This lets users model customer profiles based on historical data, but also adjust those models in reaction to real-time activity on the site.

Here’s how Kimball describes the problems WibiData addresses:

Data about users has challenges associated with it that you don’t necessarily see with other large-scale data.

• To analyze users, you need to digest large volumes of log-oriented transactional data as well as more concise profile data
• You need to serve recommendations and other derived data interactively
• There’s a mix of batch (offline) and on-the-fly calculations required to deliver recommendations at web speed

WibiData is designed to store this transactional data side-by-side with profile and other derived data attributes. Keeping data logically and physically close enables high-performance analysis of the entire data picture surrounding a user.

FoneDoktor, without Wibidata

In some cases, as FoneDoktor’s Alex Loddengaard explained in a December blog post, WibiData can obviate the need to maintain a Hadoop cluster and a separate online transaction processing system (OLTP) because WibiData provides both capabilities. It does this by using HBase as the real-time data store for transactions, and by incorporating a programming framework that’s abstracted from MapReduce so users can perform either batch or real-time analyses.

Where Avro comes in is for adding fields to data records, or adjusting schema, without affecting existing processes that have to access that data. As Kimball explains, “Does your web site track a new cookie? This can be added as a new field. But even though you start collecting that new data, your existing analysis pipelines can treat records like they always did; programs that don’t yet know about the new cookie are still compatible with both the old records already collected, and the new records with the additional field.”

Its data-management methods and machine-learning libraries for capabilities such as content recommendation make WibiData ideal for web-user data, but Kimball points out it’s also a good fit for “mobile, online gaming, healthcare, finance, and several other industries.” However, WibiData is just one of many startups looking to parlay its founders’ Hadoop expertise into a higher-level analytics product that does things Hadoop alone without requiring deep Hadoop or analytics expertise on the customer end. Good thing there’s plenty of data to go around.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013
• Scaling Hadoop clusters: the role of cluster management

The problem surfaced again Monday in the form of complaints over Forrester’s new report titled “Enterprise Hadoop Solution, Q1 2012.” InformationWeek spoke with a few vendors that didn’t like how their products were assessed, and database industry analyst Curt Monash says the report “compares apples, peaches, almonds, and peanuts.” I thought the same thing when I saw a copy of the report last week. They all focus on Hadoop, but Hortonworks is not Datameer is not HStreaming.

Allow me to explain. Hopefully, this provides a foundation for parsing what people talk about when they talk about Hadoop, and for differentiating one type of product from another. (And you can learn even more about Hadoop and how it’s used at our Structure: Data conference taking place next month in New York City.)

What Hadoop is

I went into this in more detail in a GigaOM Pro report published last March (sub req’d), but the long and short is that Hadoop is, at its core, an Apache Software Foundation project consisting of two primary subprojects — Hadoop MapReduce and the Hadoop Distributed File System. MapReduce is the parallel-processing engine that allows Hadoop to churn through large data sets in relatively short order. HDFS is the distributed file system that lets Hadoop scale across commodity servers and, importantly, store data on the compute nodes in order to boost performance (and potentially save money). These are the two must-have components for any Hadoop distribution.

There are also a number of Apache projects related to Hadoop, often built atop either Hadoop MapReduce or HDFS. These include — but are not limited to — Hive and Pig, two SQL-like query languages to provide data-warehouse-like capabilities to a Hadoop cluster, and HBase, a NoSQL database that leverages HDFS as its distributed storage engine.

Hadoop distributions

These are packaged software products that aim to ease deployment and management of Hadoop clusters compared with simply downloading the various Apache code bases and trying to cobble together a system. Presently, Cloudera, Hortonworks, MapR and EMC  all offer their own Hadoop distributions. Although they’re all unique — sometimes very unique, as with MapR’s proprietary file system — they all package a set of Hadoop projects (MapReduce, Hive, Sqoop, Pig, etc.) in a way that in theory makes them integrate more naturally, and to run both smoothly and securely.

Many Hadoop distributions integrate with various data warehouses, databases and other data-management products, with the goal of moving data between Hadoop clusters and other environments so each might process or query data stored in the other.

Hadoop management software

Just as the wording implies, Hadoop management software is designed to make it easier to manage and troubleshoot a Hadoop cluster. Such products are usually sold or offered by companies peddling Hadoop distributions, because even when commercially packaged, Hadoop is still a complex architecture and somewhat foreign to most IT personnel and products. However, third parties such as Platform Computing (now part of IBM) and Zettaset also sell software for managing Hadoop clusters, and their products are typically agnostic as to what distributions they support.

But distributions and management software are all about the infrastructure and the platform. Anyone actually wanting to use Hadoop still needs to know how to write applications that leverage the underlying architecture.

Hadoop application software (or, products that use Hadoop)

The Hadoop ecosystem gets really complex when we start looking at products that exist to help developers write Hadoop applications or otherwise analyze data stored within Hadoop in a manner other than writing traditional MapReduce jobs. These range from abstraction layers such as Karmasphere Analyst or IBM Infosphere BigInsights, to Hadapt, which offers a single-platform product fusing a SQL data warehouse with a Hadoop cluster, to HStreaming, which promises real-time processing and analytics.

The one common thing among all these products, however, is that they are not Hadoop distributions, but sit atop platform software from Hortonworks, EMC or whomever. Some products that get thrown into the Hadoop fray, such as Outerthought Lily or Drawn to Scale Spire, are essentially scale-out databases built atop HBase (which itself is a separate project built atop HDFS). The image below, from Karmasphere, gives a particularly clear map of how a Hadoop environment might look.

The applications and analytics space is probably where we’ll see the biggest influx of new companies, as writing Hadoop applications is still tough, but it’s also how companies will actually start experiencing direct business benefits. In fact, it’s these type of higher-level products that are the focal point of Accel Partners’ new big data fund.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining Hadoop: the Players, Technologies and Challenges of 2011
• A near-term outlook for big data
• Sector RoadMap: SQL-on-Hadoop platforms in 2013

Big data is hot, but infrastructure-level platforms such as Hadoop, which focus on storage and processing, still need help to take them into the mainstream. They need a killer app or two that will let companies analyze, visualize and act on all that data without hiring a team of Stanford Ph.Ds, or that will let developers write big-data apps without having to reinvent the wheel.

Here are five startups (in alphabetical order) either in stealth mode or just out of it that could help take Hadoop and its ilk to the promised land.

1. BloomReach

The stealth-mode BloomReach is taking a very targeted, very hands-free approach to big data for its customers. It’s offering a SaaS-based product that job listings say is for “helping leading online businesses uncover the highest quality, most relevant content sought by their consumers, when and where they want it.” Founded by a team with roots at Google, Cisco, Facebook and Yahoo, among other companies, BloomReach has, according to one estimate, about 160 customers — all of them among the top 10,000 websites, and most of them in the retail space. Among its core technologies and methods are Hadoop, Lucene, Monte Carlo simulations and large-scale image processing.

2. Continuuity

Continuuity, the just-launched stealth-mode startup by former Yahoo VP and chief cloud architect Todd Papaioannou, wants to make it easier to build applications that can leverage both cloud computing and big data technologies. As Papaioannou told me recently, most developers shouldn’t have to go through what Yahoo, Facebook and others did in order to write large-scale, data-driven applications. He also said “the data fabric is the next middleware” and noted that the company name is a play on “continuum.” You figure out what it’s up to.

3. Odiago

Odiago is the brainchild of Hadoop and analytics experts Christophe Bisciglia and Aaron Kimball, and aims to improve the state of web analytics. Its first product, Wibidata, which is in private beta, lets websites better analyze their user data to build more-targeted features. It’s built atop Hadoop and HBase, but also plugs into companies’ existing data-management and BI tools. Current customers include Wikipedia, RichRelevance, FoneDoktor and Atlassian (with whom it shares office space).

4. Platfora

Platfora, which launched in September with $5.7 million in funding, wants to make big data analytics accessible to the masses. Founder and CEO Ben Werther, formerly of Greenplum and NoSQL startup DataStax, told me when Platfora launched that its intuitive, visually stunning interface will make Hadoop-based analytics so easy even a history major could use it. Platfora’s product isn’t available yet, but the company is currently hiring, with an emphasis on frontend and user-experience skills.

5. SkyTree

Skytree is probably the stealthiest of the group, but it’s also is one of the more ambitious — because it’s trying to bring high-performance machine learning to mainstream companies. Machine learning is an impressive technique in which the system itself gets smarter as it digests more data, but it usually doesn’t find its way out of research environments or cutting-edge analytics teams. Skytree is putting together an impressive team, including co-founder Alexander Gray, who also teaches machine learning at Georgia Tech and spent six years at NASA’s Jet Propulsion Laboratory. The company will officially launch later this quarter.

We’ll be addressing many of the issues these companies are trying to resolve at our Structure: Data event that takes place March 21-22 in New York City. Founders from Continuuity, Odiago and Skytree will be speaking at the event, as will dozens of other data visionaries from companies such as IBM, Google, @WalmartLabs and Hortonworks.

Feature image courtesy of Flickr user jurvetson.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• A near-term outlook for big data
• How data warehousing is now a cost-effective solution for businesses