According to Quantcast VP of Research and Development Jim Kelly, the web-audience measurement specialist began working with Hadoop in 2006 and experienced problems almost from the start. However, while the early problems with HDFS might have been symptoms of its immaturity, the problems soon began centering around the two things Hadoop is supposed to be best at — size and speed. So, in 2008, Quantcast began experimenting with, and actually sponsoring, the Kosmix project.

It turns out that wasn’t a moment too soon. By 2010, after Quantcast began integrating with ad networks, its data flow really began picking up into the tens of terabytes a day range. It turned on QFS as its production Hadoop file system in 2011 and now receives about 40TB a day and processes a whopping 20 petabytes. Kelly said Quantcast has pushed 4 exabytes — or 4 billion gigabytes — through QFS since turning it on.

Faster, yes. Bigger, not so much.

At Quantcast’s scale, the problem with HDFS wasn’t so much its scalability, but the sheer size of the cluster required to handle petabyte-scale data stores. HDFS stores three copies of each piece of data to ensure they’re always available, although it tries to make up for the size issue with data locality (i.e., putting data directly on the computing nodes so it doesn’t have to traverse the network in order to be processed). Kelly thinks those techniques are relics of a bygone era.

“When HDFS [was created], disk drives and networks were tied for being the slowest things in the cluster,” he said.

Enter Reed-Solomon error correction, QFS’s chosen method for assuring reliable access to data that Kelly says actually ends up shrinking the size of Hadoop clusters while improving their performance. (It’s actually the same method used on CDs and DVDs.) Rather than storing three full versions of each file like HDFS, resulting in the need for three times more storage, QFS only needs 1.5x the raw capacity because it stripes data across nine different disk drives. Quantcast believes smaller cluster size, combined with today’s 10-gigabit networks and the ability to read and write data in parallel make QFS significantly faster than HDFS at large scale.

QFS also comes equipped with other features that Quantcast had to implement to make it production-ready. Among them: it is written in C++ and has fixed-footprint memory management; it has access control based on users and groups; and it intelligently detects node failures, as opposed to planned maintenance, and invokes data recovery accordingly.

It’s not for everyone, though

Despite its claimed improvements over HDFS though, Kelly is quick to point out that QFS is probably not the best choice for everyone. It’s really designed for Hadoop users operating at petabyte scale, who have the technical prowess to handle a migration away from HDFS, and for whom data-processing costs are hitting the six-to-seven-figure range monthly once things such as energy bills accounted for.

“If you’re cluster only has 10 disk drives,” Kelly said, “[QFS] will save you $500, which is nice but …”

Likewise, if high availability is very important, the latest version of HDFS might be preferable. “There’s a standby [in QFS]; it’s not quite as hot as theirs,” Kelly said. But availability isn’t super important to Quantcast, he said, it hasn’t had any real problems with QFS going down anyhow. When it does, it actually recovers pretty fast.

As for the rest of the file systems touting themselves as better alternatives for HDFS, Kelly didn’t have much to say. Quantcast’s efforts are focused on mega-scale Hadoop deployments, and he doesn’t see anything better for that use case. Although, he noted, Hadoop vendors probably shouldn’t get too upset over all the competition.

“I think some diversity in the ecosystem is probably not a bad thing,” he said, “and is probably a sign of healthy evolution.”

Feature image courtesy of Shutterstock user Lobke Peers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• How search can unlock the power of big data

At its core, oxdata’s flagship product, called H2O, is a statistical analysis engine that uses the Hadoop Distributed File System (HDFS) as its storage platform, but the goal is to make it as simple as using a Google service such as BigQuery. Users will interact with H2O via a simple web-search-like bar and standard R statistical-analysis syntax, but H2O will run machine-learning algorithms behind the scenes. Alternatively, users can call out to H2O from Microsoft Excel or the RStudio integrated development environment using a REST API.

Although BigQuery is a SQL service hosted by Google, 0xdata follows a similar theory on simplicity.

However they choose to leverage the product, Ambati said, the scale of the underlying data and the complexity of running advanced analysis are details that need to be hidden. It’s the same theory that underlies Platfora, the company Ambati co-founded last year with his former DataStax colleague Ben Werther, although their approaches appear to be different. Whereas Platfora is trying to disrupt the data warehouse market by building a next-generation user experience atop Hadoop, 0xdata is trying to change the way users interact with popular statistical software such as R.

But either way, Ambati says of new data-analysis products, “[There are] no bragging rights for making it simple. If you don’t do that, you won’t be able to go forward.”

oxdata is also putting a focus on speed, both in terms of how fast it processes data and how fast it lets users react. Google search changed our thinking around how many questions people can ask successively, Ambati explained, and data analysts should have the same experience. That’s why H2O provides approximate results at every step in the analysis process. Rather than wait for the entire job to run and the exact results to be computed, users can get a general idea of results and kill the job and start over quicker if they’re completely outside the expected range.

But it will be a while before the public gets a chance to see whether H2O lives up to its promises. Ambati said the product is just four months into development and won’t have its first set of algorithms available for another few months. His team of eight engineers has “built a lot of cool stuff,” but now it needs to round out the process and turn its code for H2O into an actual product.

Still, having decided to tackle data as a system, Ambati and his team are having a lot of fun. “We are live-and-die-with-infrastructure people,” he said, but for a bunch of folks who spent a lot of time learning math, it’s like going back to the their days as computer science students.

Feature image courtesy of Shutterstock user Bruce Rolff.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The importance of putting the U and I in visualization
• 2012: The Hadoop infrastructure market booms

In complaints filed this week, a Delaware-based shell called Parallel Iron claims Facebook and LinkedIn violated its patents by using HDFS. Parallel Iron has already filed suits against Amazon, Oracle and other firms for using the HDFS technology that lets users store huge quantities of data on clusters of commodity servers.

Hadoop has been built by a large network of contributors, including individual developers and large companies like Yahoo and is an Apache Software Foundation project. HDFS, its storage component, was based on Google’s Google File System. Parallel Iron’s patent complaints, however, say the whole system was made possible by four men:

In this technological age, we take for granted the ability to access tremendous amounts of data through our computers and the Internet, a process that seems effortless and unremarkable. But this apparent effortlessness is an illusion, made possible only by technological wizardry. …  It was made possible by the innovations of technological pioneers like Melvin James Bullen, Steven Louis Dodd, William Thomas Lynch, and David James Herbison.

The four men obtained three patents for “methods and systems for a storage system” in 2007, 2009 and 2011 (click the dates to see them). They assigned the patents to an LLC called Ring Technology Enterprises which appears to have been a predecessor shell company to Parallel Iron that filed lawsuits in the Eastern District of Texas.

Such companies, known as patent trolls, have come under fire as critics accuse of them gaming the patent system in order to extort money from companies that create real products and services. Recently, trolls have begun stalking promising start-ups (like travel site Hipmunk and handcraft marketplace Etsy) and suing them as soon they receive funding.

The toll of patent trolls on innovation has received special attention this month as two academics released a study concluding that trolls cost the economy $29 billion in direct costs every year. Meanwhile, a famous judge called the patent system “dysfunctional” and threw out a long-awaited smartphone trial between Google and Apple. This week, the same judge wrote an editorial in the Atlantic saying many industries don’t need patents in the first place and suggesting that trolls should have to actually use the patents that are the basis of their lawsuits.

As Congress has been slow in fixing the patent troll problem, some companies like Twitter are exploring their own solutions like pledging not to use their patents in an offensive manner.

Parallel Iron’s attorney, Richard Kirk, did not immediately return a request for comment.

Here’s a copy of the complaint against Facebook:

Parallel Iron v Facebook

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Amazon’s DynamoDB: rattling the cloud market
• How search can unlock the power of big data

Within the Apache Software Foundation, HDFS is always improving in terms of performance and availability. Honestly, it’s probably fine for the majority of Hadoop workloads that are running in pilot projects, skunkworks projects or generally non-demanding environments. And technologies such as HBase that are built atop HDFS speak to its versatility as storage system even for non-MapReduce applications.

But if the growing number of options for replacing HDFS signifies anything, it’s that HDFS isn’t quite where it needs to be. Some Hadoop users have strict demands around performance, availability and enterprise-grade features, while others aren’t keen of its direct-attached storage (DAS) architecture. Concerns around availability might be especially valid for anyone (read “almost everyone”) who’s using an older version of Hadoop without the High Availability NameNode. Here are eight products and projects whose proprietors argue can deliver what HDFS can’t:

Cassandra (DataStax)

Not a file system at all but an open source, NoSQL key-value store, Cassandra has become a viable alternative to HDFS for web applications that rely on fast data access. DataStax, a startup commercializing the Cassandra database, has fused Hadoop atop Cassandra to provide web applications fast access to data processed by Hadoop, and Hadoop fast access to data streaming into Cassandra from web users.

Ceph

Ceph is an open source, multi-pronged storage system that was recently  commercialized by a startup called Inktank. Among its features is a high-performance parallel file system that some think makes it a candidate for replacing HDFS (and then some) in Hadoop environments. Indeed, some researchers started looking at this possibility as far back as 2010.

Dispersed Storage Network (Cleversafe)

Cleversafe got into the HDFS-replacement business on Monday, announcing a product that will fuse Hadoop MapReduce with the company’s Dispersed Storage Network system. By fully distributing metadata across the cluster (instead of relying on a single NameNode) and not relying on replication, Cleversafe says it’s much faster, more reliable and scalable than HDFS.

GPFS (IBM)

IBM has been selling its General Parallel File System to high-performance computing customers for years (including within some of the world’s fastest supercomputers), and in 2010 it tuned GPFS for Hadoop. IBM claims the GPFS-SNC (Shared Nothing Cluster) edition is so much faster than Hadoop in part because it runs at the kernel level as opposed to atop the OS like HDFS.

Isilon (EMC)

EMC has offered its own Hadoop distributions for more than a year, but in January 2012 it unveiled a new method for making HDFS enterprise-class — replace it with EMC Isilon’s OneFS file system. Technically, as EMC’s Chuck Hollis explained at the time, because Isilon can read NFS, CIFS and HDFS protocols, a single Isilon NAS system can serve to intake, process and analyze data.

Lustre

Lustre is a an open source high-performance file system that some claim can make for an HDFS alternative where performance is a major concern. Truth be told, I haven’t heard of this combination running anywhere in the wild, but HPC storage provider Xyratex wrote a paper on the combination in 2011, claiming a Lustre-based cluster (even with InfiniBand) will be faster and cheaper than an HDFS-based cluster.

MapR File System

The MapR File System is probably the best-known HDFS alternative, as it’s the basis of MapR’s increasingly popular — and well-funded — Hadoop distribution. Not only does MapR claim its file system is two to five times faster than HDFS on average (although, really, up to 20 times faster), but it has features such as mirroring, snapshots and high availability that enterprise customers love.

NetApp Open Solution for Hadoop

OK, the NetApp Open Solution for Hadoop isn’t so much an HDFS replacement as it is an HDFS improvement, according to NetApp and early partner Cloudera. The offering still relies on HDFS, but it reenvisions the physical Hadoop architecture by putting HDFS on a RAID array. This, NetApp claims, means faster, more reliable and more secure Hadoop jobs.

This might be a good place to say rest in peace to two other HDFS alternatives that are effectively no longer with us — KosmosFS (aka CloudStore) and Appistry CloudIQ Storage. The former was created by Kosmix (since bought by @WalmartLabs) and released to the open source world in 2007, but no longer has an active community. The latter was an attempt by Appistry in 2010 to get a piece of the Hadoop pie with its computational storage technology, but the company has since switched its focus from selling the technology to providing high-performance computing services based on it.

Feature image courtesy of Shutterstock user Panos Karapanagiotis.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• Amazon’s DynamoDB: rattling the cloud market

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Overview, Q2 2010

For those unfamiliar with the availability problem Facebook solved with AvatarNode, here’s the 10,000-foot explanation: The NameNode service in Hadoop’s architecture handles all metadata operations with the Hadoop Distributed File System, but it also just runs on a single node. If that node goes down, so does, for all intents and purposes, Hadoop because nothing that relies on HDFS will run properly.

As Ryan explains, Facebook began building AvatarNode about two years ago (hence its James Cameron-inspired name) and it’s now in production. Put simply, AvatarNode replaces the NameNode with a two-node architecture in which one acts as a standby version if the other goes down. Currently, the failover process is manual but, Ryan writes, “we’re working to improve AvatarNode further and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover.”

AvatarNode isn’t a panacea for Hadoop availability, however. Ryan notes that only 10 percent of Facebook’s unplanned downtime would have been preventable with AvatarNode in place, but the architecture will allow Facebook to eliminate an estimated 50 percent of future planned downtime.

Facebook isn’t the only company to solve this problem, by the way. Appistry (which has since changed its business focus) released a fully distributed file system a couple years ago, and MapR’s Hadoop distribution also provides a highly available file system. In Apache Hadoop version 2.0, which underpins the latest version of Cloudera’s distribution, the NameNode is also eliminated as a single point of failure.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• How search can unlock the power of big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Infrastructure Q2: Big data and PaaS gain more momentum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why service providers matter for the future of big data
• Infrastructure Q2: Big data and PaaS gain more momentum
• 2012: The Hadoop infrastructure market booms

The problem surfaced again Monday in the form of complaints over Forrester’s new report titled “Enterprise Hadoop Solution, Q1 2012.” InformationWeek spoke with a few vendors that didn’t like how their products were assessed, and database industry analyst Curt Monash says the report “compares apples, peaches, almonds, and peanuts.” I thought the same thing when I saw a copy of the report last week. They all focus on Hadoop, but Hortonworks is not Datameer is not HStreaming.

Allow me to explain. Hopefully, this provides a foundation for parsing what people talk about when they talk about Hadoop, and for differentiating one type of product from another. (And you can learn even more about Hadoop and how it’s used at our Structure: Data conference taking place next month in New York City.)

What Hadoop is

I went into this in more detail in a GigaOM Pro report published last March (sub req’d), but the long and short is that Hadoop is, at its core, an Apache Software Foundation project consisting of two primary subprojects — Hadoop MapReduce and the Hadoop Distributed File System. MapReduce is the parallel-processing engine that allows Hadoop to churn through large data sets in relatively short order. HDFS is the distributed file system that lets Hadoop scale across commodity servers and, importantly, store data on the compute nodes in order to boost performance (and potentially save money). These are the two must-have components for any Hadoop distribution.

There are also a number of Apache projects related to Hadoop, often built atop either Hadoop MapReduce or HDFS. These include — but are not limited to — Hive and Pig, two SQL-like query languages to provide data-warehouse-like capabilities to a Hadoop cluster, and HBase, a NoSQL database that leverages HDFS as its distributed storage engine.

Hadoop distributions

These are packaged software products that aim to ease deployment and management of Hadoop clusters compared with simply downloading the various Apache code bases and trying to cobble together a system. Presently, Cloudera, Hortonworks, MapR and EMC  all offer their own Hadoop distributions. Although they’re all unique — sometimes very unique, as with MapR’s proprietary file system — they all package a set of Hadoop projects (MapReduce, Hive, Sqoop, Pig, etc.) in a way that in theory makes them integrate more naturally, and to run both smoothly and securely.

Many Hadoop distributions integrate with various data warehouses, databases and other data-management products, with the goal of moving data between Hadoop clusters and other environments so each might process or query data stored in the other.

Hadoop management software

Just as the wording implies, Hadoop management software is designed to make it easier to manage and troubleshoot a Hadoop cluster. Such products are usually sold or offered by companies peddling Hadoop distributions, because even when commercially packaged, Hadoop is still a complex architecture and somewhat foreign to most IT personnel and products. However, third parties such as Platform Computing (now part of IBM) and Zettaset also sell software for managing Hadoop clusters, and their products are typically agnostic as to what distributions they support.

But distributions and management software are all about the infrastructure and the platform. Anyone actually wanting to use Hadoop still needs to know how to write applications that leverage the underlying architecture.

Hadoop application software (or, products that use Hadoop)

The Hadoop ecosystem gets really complex when we start looking at products that exist to help developers write Hadoop applications or otherwise analyze data stored within Hadoop in a manner other than writing traditional MapReduce jobs. These range from abstraction layers such as Karmasphere Analyst or IBM Infosphere BigInsights, to Hadapt, which offers a single-platform product fusing a SQL data warehouse with a Hadoop cluster, to HStreaming, which promises real-time processing and analytics.

The one common thing among all these products, however, is that they are not Hadoop distributions, but sit atop platform software from Hortonworks, EMC or whomever. Some products that get thrown into the Hadoop fray, such as Outerthought Lily or Drawn to Scale Spire, are essentially scale-out databases built atop HBase (which itself is a separate project built atop HDFS). The image below, from Karmasphere, gives a particularly clear map of how a Hadoop environment might look.

The applications and analytics space is probably where we’ll see the biggest influx of new companies, as writing Hadoop applications is still tough, but it’s also how companies will actually start experiencing direct business benefits. In fact, it’s these type of higher-level products that are the focal point of Accel Partners’ new big data fund.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining Hadoop: the Players, Technologies and Challenges of 2011
• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond