Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

In a conference held in San Francisco, VP and General Manager of Intel’s Datacenter Software Division Boyd Davis explained Intel’s history in Hadoop that stretches back to 2009 and stressed that Intel is going to share some aspects of its Hadoop distribution, but not all. Intel has a distribution of Hadoop it has released in China, but today it’s bringing it to the United States Intel’s version of the Hadoop distribution uses Hadoop 2.0 and YARN, which is a cutting-edge version of  platform compared with what most Hadoop users have deployed thus far.

Why Intel wants to push its own version of Hadoop

Boyd introduced partners such as and Cisco, which has tuned the Intel Hadoop distribution for its own servers. Intel also hosted a panel that included executives from SAP, Red Hat and Savvis to discuss the challenges of big data and the promise of Hadoop and big data.

Davis was up front about Intel’s rationale for releasing its own distribution, namely that it was worried about the fragmentation and possible uncertainty associated with current Hadoop distributions. That could be read as a dig against the many startups already offering Hadoop distributions, all of which are slightly different (of course, Intel’s will be slightly different, too). Like all of the existing players such as Cloudera and MapR, Intel will open source certain aspects of its distribution, but will also keep software to itself.

Inside the data center, it’s no longer just web servers that matter

For example, Davis stressed that Intel will not share its management and monitoring software, which could be highly valuable for enterprise customers. The Intel software could coordinate with Intel’s data center management software and make managing a variety of workloads easier. And hidden in that coordination might be one Intel’s aims in pushing its own version of Hadoop — the threat of ARM chips used in Hadoop clusters.

Dell, Calxeda and others are evaluating the use of lower-performance, lower-power chips in Hadoop clusters, a market Intel would hate to cede in the data center as data grows and analytics becomes more important. To that end, Intel has also optimized its Hadoop distribution for solid-state drives, something that other Hadoop companies haven’t done so far.

When asked about Atom and the use of lower-performance processors for Hadoop, Davis noted that while people are using lower-end processors for Hadoop , but that those uses tend to have slower networking. Davis says that when you combine high-end processors with 10 gigabit Ethernet and Hadoop, customers get the performance that they want.

So while Intel may tout stability and consistency as the reason for it’s decision to become a major player in the software market for big data, it’s also driven by the changes in the data center that threaten the grip Intel has on the hardware inside the data center. The cloud and big data has changed the workloads and hardware requirements for the data center and Intel is playing the long game in trying to release software that can be tuned to its chips.

The Hadoop drama isn’t over yet

Intel isn’t the only big vendor touting its own homegrown version of Hadoop. On Monday, EMC’s Greenplum division announced an entirely revamped version of its Hadoop distribution that’s merged with it’s flagship analytic SQL database. These big companies have big existing businesses to protect and lots of resources to put into doing it. As my colleague Derrick Harris wrote on the EMC news:

Looking past his competitive boasting, though, it’s easy to see [Greenplum's Scott] Yara’s greater point when you ask him what all this Hadoop talks means for the data warehouse business on which Greenplum was built. He points to the mainframe business that fell from its high perch decades ago but still drives billions a year in revenue. A single MPP database system is still faster on certain workloads than SQL on Hadoop, but that gap will close over time and “I do think the center of gravity will move toward HDFS,” he said.

Hadoop is a juggernaut when it comes to big data. Intel is a juggernaut when it comes to data center infrastructure. Its decision to enter into the open source software market is a big one for the chip company, for the Hadoop ecosystem and for the myriad startups playing in this space. It’s a topic we’ll explore more during our Structure Data conference in New York on March 20 and 21.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Why service providers matter for the future of big data

The combination should  make it easier to integrate data from SQL Server and Hadoop and to funnel all that into Excel for charting and pivoting and all the tasks Excel is good at, Connolly added.

He stressed that this means the very same Apache Hadoop distribution will run on Linux and Windows. An analogous Hortonworks Data Platform for Windows Azure is still in the works.

Microsoft opted to work with Hortonworks rather than to continue its own “Dryad” project, as GigaOM’s Derrick Harris reported a year ago. Those with long memories will recall this isn’t the first time that Microsoft relied on outside expertise for database work. The guts of early SQL Server came to the company via Sybase.

The intersection of structured SQL and  unstructured Hadoop universes is indeed a hotspot, as Derrick Harris reported last week, with companies including Hadoop rivals Cloudera and EMC Greenplum all working that fertile terrain. That means Hortonworks/Microsoft face stiff competition. This topic, along with real-time data tracking, will be discussed at GigaOM’s Structure Data conference in New York on March 20-21.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Everything, of course, is relative and we might never see a Hadoop vendor reach the size of a database company such as Oracle with more than 100,000 employees and tens of billions in annual revenue. After all, Hadoop is a new technology for most companies, so it’s not really moving in on an already lucrative market and stealing budgetary dollars from incumbents. Further — and possibly more importantly — the core Hadoop technology is free and open source, meaning there are lots of unpaid downloads so money comes from services, support and large enterprises willing to buy software licenses for value-added products.

Money

Here’s a chart showing how much money Hadoop-based companies have raised thus far (although the grand total will likely rise by at least $10 million next week). Keep in mind, Cloudera only launched in 2009 and Hortonworks launched in June 2011. And these aren’t companies that merely bury Hadoop under an application or can connect their technologies to it — these are companies either selling Hadoop or applications designed specifically for it.

(To view the original, interactive chart, click here.)

In terms of revenue, one might look to a May 2012 report by research from IDC estimating the size of the Hadoop ecosystem to be around $77 million, growing to $813 million by 2016. Those are both impressive numbers, but they might actually be short-changing reality. For one, as I noted at the time, the authors attributed almost no revenue to Amazon Web Services’ Elastic MapReduce service, which is almost certainly generating at least a few million in revenue each year.

Speaking to me in June, Cloudera CEO Mike Olson also took issue with the number, claiming it didn’t even take Cloudera’s revenue into account — which seems entirely possible considering the business Cloudera is doing. I’ve heard from reliable sources that Cloudera is doing very well and is on track to do about $100 million in revenue this year, very possibly more. And as early as April 2011, Cloudera executives were touting that software license revenue had already surpassed services revenue (although it’s arguable whether that will, or even has to, remain the case).

More anecdotally, I’ve heard from several sources that Hortonworks has already declined at least one potentially appealing acquisition offer. That it wouldn’t sell isn’t surprising: sources say the company is valued at $225 million after its last round of funding and is looking to raise more money. And although it just released its first product in June, the company has impressive and potentially lucrative partnerships in place with Microsoft, Teradata, Rackspace, VMware and other large vendors.

MapR, the proprietary thorn in the sides of both Cloudera and Hortonworks, appears to be doing quite well, too. Vice President of Marketing Jack Norris told me in June that his company had higher license revenue than many would expect and predicted that deals with Amazon Web Services and Google Compute Engine would help the company become “the license revenue leader within the next quarter.”

Former Cloudera VP of Technology Solutions Omer Trajan, who just left to join HBase-centric startup WibiData, shared some insights with me from his days at Cloudera that seem to back up vendor confidence. He said most mature production clusters (excluding monster users such as Facebook) consist of about 200 nodes, and many double in size after the first year. That’s part of the reason Cloudera grew in size about 10x during the three years he was there.

“It has definitely been a rocket ship,” he said. “… You just strap in and hope you make it up.”

Interest is only picking up, too: “There are more people that have started big data projects in the past six months than have big big data projects running [in production],” Trajman said.

People

It’s probably not accurate to call companies such as Cloudera, Hortonworks and MapR startups anymore, and we might start to see signs of this shift in personnel moves. Here’s how big they are and expect to become:

• Cloudera: More than 300 employees globally and growing, especially in the sales department.
• Hortonworks: 145 employees as of late October and hiring a person per day, on average, through the end of 2012.
• MapR: More than 125 employees, mostly in technical and engineering positions; starting to build sales team and looks to more than double headcount in 2013.

While Cloudera and Hortonworks, for example, are still young, nimble and agile enough to lure a fair amount of talent from now-officially large enterprises such as VMware, their employees who joined on early and really love the startup life might not stick around.

Trajman’s new home, WibiData, is a fine example of this. It was launched last year by former Cloudera employees Christophe Bisciglia (who actually co-founded Cloudera) and Aaron Kimball to help companies build behavioral-analysis applications on top of Hadoop.

(Maybe there’s a Cloudera mafia shaping up: WibiData’s officemates — MemCachier and Thanx — both count former Cloudera employees as key members or founders of their teams, as does HBase-centric startup Drawn to Scale.)

Trajman, who was one of the first couple dozen employees at Cloudera (and who previously joined Vertica at around the same stage in its growth) told me he likes the rush of getting in the the ground level of new technologies and helping companies do something really new. While he enjoyed establishing and implementing some the the core foundational use cases for Hadoop (e.g., ETL and data exploration) with Cloudera’s early customers, that’s still much of what Cloudera provides to customers because it’s so difficult to build higher-level and higher-value applications at the infrastructural level where Cloudera operates.

“For me, it was very personal in terms of the impact I wanted to have,” Trajman said. At WibiData, he can help users who have the infrastructure part resolved and now want to develop applications that make data analysis a core part of their businesses. Where there’s a focus on innovation, he said, that’s where the innovators go.

This isn’t a bad thing, it’s just a side effect of growth — and when employees stay and innovate in the Hadoop space, it just creates a bigger pie for everyone to share.

Feature image courtesy of Shutterstock user GuskovaNatalia.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Why service providers matter for the future of big data
• A near-term outlook for big data

From batch processing to data interaction

The business world regularly laments the circumstances that spurred Impala’s creation. I summed them up last week and again yesterday when reporting similar products from startups Hadapt and Platfora, but the gist is that although Hadoop is more scalable and more flexible than traditional data warehouses or analytic databases, it’s also slower, harder to learn and designed for batch processing an entire data set rather than interactively querying a data set. Until now, the common methods for querying Hadoop were to use a custom-built language such as Hive, or to transport data to a data warehouse from Hadoop and then analyze it using traditional business intelligence software.

However, Cloudera’s Cloud VP of Products Charles Zedlewski was quick to point out during a recent conversation that Impala isn’t a replacement for other BI tools, just a new data source into which they can connect. If anything, it’s a replacement for Hive, which Facebook built to bring data warehouse capabilities to Hadoop, but which wasn’t really developed for public consumption as a software product. For the sake of uniformity, Impala actually uses the same SQL set as Hive, but is on average 10 times faster thanks to its purpose-built query engine that foregoes reliance on MapReduce. Small queries, Zedlewski said, can run in less than a second.

Impala has been in the making for almost two years, and Cloudera “took a a lot of pains to stitch this really well in with the rest of the Hadoop stack,” Zedlewski said. Users still store data in the Hadoop Distributed File System of the HBase database, and they can still store whatever types of structured, semi-structured on unstructured data they please. Impala uses the same metadata as the other Hadoop components, the same drivers and — like almost everything else in the Hadoop world — is open source under the Apache Software Foundation license.

Unlike some other Hadoop startups, though, Cloudera isn’t interested in selling BI or other analytic applications. Impala (which is called Real-Time Query for customers who pay for support) is the execution engine, but it still relies on software from Cloudera partners such as Tableau, QlikTech and MicroStrategy in order to ask questions and visualize the results. “We’re sticking to our knitting as a platform vendor,” said Zedlewski, echoing a sentiment on which his boss, Cloudera CEO Mike Olson, has been bullish for years.

Different strokes move the world

I can’t underscore enough how critical all of this innovation is for Hadoop, which in order to add substance to its unparalleled hype needed to become far more useful to far more users. But the sudden shift from Hadoop as a batch-processing engine built on MapReduce into an ad hoc SQL querying engine might leave industry analysts and even Hadoop users scratching their heads.

Cloudera, now with more than 300 employees and annual revenue rumored to be in hundreds of millions, is the 800-pound gorilla in the Hadoop market, and its implementation of Impala has to make it look even better for prospective customers. But Cloudera doesn’t have this space to itself. Assuming your goal is to use Hadoop as the platform for running SQL queries (as opposed to, for example, using it for ETL before putting it in an in-memory system), there are plenty of choices on the table. And everyone’s approach is different.

For starters, bitter distribution-level rival MapR announced in August that it’s leading an open source project called Drill that provides essentially the same functionality as Impala. MapR is getting a lot of love from Hadoop users right now, and a future implementation of Drill into its product lineup would add even more legitimacy. Not wanting to cede the innovation edge to Cloudera of MapR, one has to suspect Yahoo spinoff Hortonworks will also get into the query engine game at some point. (We’ll leave the debate over whether the myriad different flavors of Hadoop constitute the beginning of a community fracture for another day.)

Like Cloudera, however, if MapR and Hortonworks decide to integrate query engines in their products, they’ll likely rely on application providers to deliver the user experience on top. For better or worse, that presently means reliance on legacy vendors until startups can get familiar with the source code and start building BI products designed to take advantage of the new capabilities. When asked about Impala as a technology for disrupting the traditional data warehouse market, Cloudera’s Zedlewski noted that existing products are often very good at what they do.

“I think it’s highly unlikely that something like Impala would really be considered an alternative of that,” he said. Those vendors don’t seem to think so either, as companies like Teradata and EMC Greenplum (e emc) are telling always-improving stories about integrating their existing product lines with Hadoop.

Running a sentiment analysis in Tableau with Hadapt

On the other end of the spectrum are startups such as Hadapt, Platfora and Birst, which have built Hadoop-based query engines on their own, independent of loyalty to any particular Hadoop distribution. These companies have a lot of smart people on board, and their technologies are for real. Platfora CEO Ben Werther, in particular, makes no bones about his goal of unseating the BI incumbents with analytics applications built from the ground up to analyze big data stored in Hadoop.

Similar, although not necessarily competitive, technologies include Spire (from Drawn to Scale) and Splice Machine. Both support some level of SQL querying and/or BI integration, although their real value comes in leveraging HBase to provide transactional capabilities that analytic databases aren’t designed to do.

Even though all these choices and approaches might add to the confusion over how to use Hadoop and which products to choose, the result is a net gain for Hadoop as the de facto platform for big data environments even in the face of some alternative approaches. It has changed from a batch system to an interactive query engine pretty much overnight, so although he wouldn’t comment on the competition, Zedlewski wasn’t just blowing vendor smoke when told me, “I would argue Impala is a proof point that Hadoop as a platform has an ability to grow that no other data management platform has.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Why service providers matter for the future of big data
• The importance of putting the U and I in visualization

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises

As with any new technology, the big question surrounding Hadoop as a viable market is whether enterprises will adopt it. The answer seems to be a resounding “Yes.” Already, Hortonworks CEO Rob Bearden told me, “We are seeing Hadoop in almost every Fortune 500 in either a proof of concept or a pilot.” Bearden doesn’t mean that his company has accounts with everyone in the Fortune 500, though, just that the majority of those companies are looking into Hadoop.

Cloudera, the first company to commercialize Hadoop (all the way back in 2008), certainly has a lot of those premier accounts. Cloudera CEO Mike Olson says interest in his company’s software and services is “absolutely skyrocketing” and it has many more deals in the pipeline that it ever has before. That’s on top of the big deployments (such as those at Nokia, Samsung and Chevron) his company already has in place.

Even MapR — a Hadoop startup that hasn’t been in the public eye as long as Cloudera and doesn’t have Yahoo roots to tout like Hortonworks does — claims to be killing it. It has flagship customers such as comScore and Boeing, as well as an OEM deal with Hadoop frenemy EMC Greenplum that MapR VP of Marketing Jack Norris told me is driving a lot of deals. EMC resells MapR’s M5 Hadoop distribution under the EMC Greenplum Hadoop MR Edition moniker.

That MapR is able to sell licenses for Hadoop — something most of its competitors give away (even MapR has a free version called M3) — says a lot about demand for Hadoop. “My guess is that we’ll be the license revenue leader within the next quarter,” Norris said. “We have higher M5 licensing and use than you would expect.”

But is Hadoop a bubble?

However, despite all the enterprise interest in Hadoop, some critics worry that it’s an overhyped technology that is bound to disappoint companies that put too much stock in it.

It’s easy to see where someone would get that idea considering how “Hadoop” rolls off the tongue as soon as a discussion turns to big data or analytics. And then there’s the money: Hadoop distribution vendors such as Cloudera, Hortonworks and MapR raise venture capital in increments of $10 million, and it seems as if every startup claiming some connection to Hadoop is able to raise at least a few million.

It all seems too good to be true. We’ve seen this story play out before with technologies that never really caught on (such as virtual desktops) and industries that collapsed despite the promise of a technological savior (think Java and the dot.com era). With many of those Hadoop installations still in the pilot phase, there’s still time for the companies testing it out to back away when it doesn’t pan out.

Nope. Here’s why.

But that skepticism appears misplaced when it comes to Hadoop, which has everything going in its favor right now. At the foundational level, where even Hortonworks’ Bearden acknowledges Hadoop “is not [yet] 100 percent intuitive,” the story is getting better. As it gets easier to deploy and manage, IT departments tasked with running Hadoop clusters are going to put up less of a fight.

Rob Bearden

And it is getting easier. Reference architectures? Check. Cluster management software? Check. Preconfigured software-hardware stacks? Check. “We have to evolve Hadoop to become an enterprise data platform,” Bearden said, and all these things — along with buy-in from the world’s largest IT companies — will help make that happen.

Oh, and now VMware wants to make Hadoop run on virtual machines to help make it more resource-efficient and dynamic. For startups, Hadoop is available in myriad formats as a cloud service, which means teams with small IT teams or budgets don’t need to own or manage a cluster at all.

Actually running analytics jobs is also getting a lot easier, especially for companies that want to extend their current practices into bigger, badder datasets. Basic analytic functions are becoming child’s play thanks to Hadoop-focused startups such as Karmasphere, Datameer and Platfora. Nearly every analytic database and business intelligence product on the planet also now connects with Hadoop. So does Microsoft Excel, an integration Cloudera’s Olson said “is going to make the the biggest change in Hadoop [adoption], generally.”

But it’s really about apps

Probably the most exciting sign of Hadoop’s prospects, though, is the number of entirely new applications it’s enabling for companies creative enough to spot the opportunities. I spent two days at Hadoop Summit this week, and while talks by Twitter and Facebook stole the show, I thought some of the most interesting (in theory, if not in practice) we’re around using Hadoop to do things like improve online education or search online forums that often house the only available information on super-niche topics.

We’ve covered many more on GigaOM through the years, ranging from better targeted advertising to better customer service to more-intelligent health care. And then there are tools such as Spire, a high-performance SQL database from startup Drawn to Scale that’s based on HBase, an open source database built atop the Hadoop Distributed File System.

Olson thinks the availability of applications — especially those built by sofware vendors and targeting specific uses within specific industries — will spur a flood of Hadoop adoption. For example, he said, an application for financial risk analysis “will be very easy to sell into hedge funds.” And as Olson told entrepreneurs considering building Hadoop-based apps at our Structure: Data conference in March, “Call me, I’ll connect you with funding. The money is out there.”

I have no reason to doubt it is. Hadoop Summit brought together about 2,200 people of all stripes that are working with this six-year-old technology in some manner. Earlier in June, Cloudera put on a product launch party in San Francisco that rivaled any IT event I’ve ever seen in terms of sheer swankiness. If Hadoop isn’t poised to become a multi-billion-dollar market very soon, it’s putting on one heck of a facade.

Feature image courtesy of Shutterstock user Carlos Caetano; Mike Olson photo by Pinar Ozger.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• Takeaways from the second quarter in cloud and data
• Why service providers matter for the future of big data

In terms of product, the Hortonworks Data Platform is about what was advertised when the company first unveiled it in November. The major difference from other commercial distributions, such as Cloudera, EMC Greenplum and MapR is that Hortonwork uses Apache Ambari to configure and manage clusters; HCatalog as a metadata service to connect with relational database products; and incorporates Talend’s Open Studio as a tool for graphically integrating datasets and composing workflows.

The Hortonworks Data Platform HA distribution architecture, however, is a bit more intriguing. Technically, it works by running important Hadoop services such as NameNode and JobTracker and Oozie on virtual machines. If the physical server or VM on which a service is running fails, the product automatically moves the service to another box.

The other commercial Hadoop distributions all offer fault tolerance — at least for the Hadoop Distributed File System (which is where the NameNode resides) — but they rely on different approaches to get there. Cloudera, for example, is built on the new Hadoop 2.0 version (Hortonworks uses the tried-and-true Hadoop 1.0), while MapR uses a proprietary file system. Hortonworks, as is its business plan, will contribute the code for its HA version solution back into the Apache Hadoop project.

“The next stage,” Hortonworks Chief Products Officer Ari Zilka told me, “is to run the whole cluster in a virtual environment.” Doing that without sacrificing processing performance will be the trick.

Where the real intrigue comes is that the Hortonworks Data Platform HA edition will be available through VMware. The work with VMware represents the furtherance of a unique partner strategy in which Hortonworks works closely with technology partners such as VMware, Microsoft and Teradata to develop products that leverage Hadoop while being more than mere integrations. Cloudera has more than 200 partners, for example, but at least some of Hortonworks’ partnerships appear much tighter.

Finally, at least, the market for Hadoop distributions appears complete. There are five rather distinct offerings from five rather distinct providers — Cloudera, EMC Greenplum, Hortonworks, IBM and MapR (six if you include Amazon’s Elastic MapReduce cloud service) — and each has its merits. We’ll see whose technologies and business models win the day as the enterprise world gets set to start investing in Hadoop in a major way.

Feature image courtesy of Shutterstock user Colin Edwards Photography.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• Infrastructure Q1: Cloud and big data woo enterprises
• Why service providers matter for the future of big data

Apache Ambari

Apache Ambari is an open source project for monitoring, administration and lifecycle management for Hadoop. It’s also the project that Hortonworks has chosen as the management component for the Hortonworks Data Platform. Ambari works with Hadoop MapReduce, HDFS, HBase, Pig, Hive, HCatalog and Zookeeper.

Apache Mesos

Apache Mesos is a cluster manager that lets users run multiple Hadoop jobs, or other high-performance applications, on the same cluster at the same time. According to Twitter Open Source Manager Chris Aniszczyk, Mesos “runs on hundreds of production machines and makes it easier to execute jobs that do everything from running services to handling our analytics workload.”

Platform MapReduce

Platform MapReduce is high-performance computing expert Platform Computing’s entre into the big data space. It’s a runtime environment that supports a variety of MapReduce applications and file systems, not just those directly associated with Hadoop, and is tuned for enterprise-class performance and reliability. Platform, now part of IBM, built a respectable business managing clusters for large financial services institutions.

StackIQ Rocks+ Big Data

StackIQ Rock+ Big Data is a commercial distribution of the Rocks cluster management software that the company has beefed up to also support Apache Hadoop. Rocks+ supports the Apache, Cloudera, Hortonworks and MapR distributions, and handles the entire process from configuring bare metal servers to managing an operational Hadoop cluster.

Zettaset Orchestrator

Zettaset Orchestrator is an end-to-end Hadoop management product that supports multiple Hadoop distributions. Zettaset touts Orchestrator’s UI-based experience and its ability to handle what the company calls MAAPS — management, availability, automation, provisioning and security. At least one large company, Zions Bancorporation, is a Zettaset customer.

If there are more Hadoop management tools floating around, please let me know in the comments.

Feature image courtesy of Shutterstock user .shock.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Scaling Hadoop clusters: the role of cluster management
• 2012: The Hadoop infrastructure market booms
• A near-term outlook for big data