7 Comments

Summary:

Hadoop, Cassandra, HBase, Hypertable, Open Neptune… these are some open source projects that are being pursued by web technologists in order to deal with explosion of digital data in a post-terabyte world. The traditional way to deal with unstructured data isn’t working. What we need is a structured means of finding, accessing, and retrieving files and objects.

The storage industry has been pumping up the growth of unstructured data for years. But unstructured data without a structured means of finding, accessing, and retrieving files and objects is basically useless. Many storage administrators are now struggling, since organizing unstructured data after the fact can be complex and difficult to scale. But that is changing with a few relatively new open source offerings.

Typically, “unstructured” refers to a mix of file types that get amalgamated into one giant data store. For example, productivity suite files (documents, spreadsheets, presentations), music files, videos and photos fall into this unstructured category. With the availability of lower-cost storage devices like SATA drives, it became economically feasible for companies to keep more and more files online. And with the surge in freely-hosted, consumer-generated content, a market was born for file systems that could scale to nearly infinite capacity. Global namespaces and clustered file systems addressed the desire to have a single data store, but often at other, hidden costs.

Think about it this way: With a single global data store comes a single global index. Combine more files with more simultaneous requests and you have a bottleneck. The system cannot find the files quickly enough, because the index gets overloaded. To address this issue, many system architects deploy a database like MySQL to serve as a separate reference for the file system. By assigning unique identification numbers to each file, and mapping that to a specific location on the file system, application requests using unique IDs can go directly to the file, bypassing directory crawls, and boosting performance. But even that solution can only go so far.

Scaling MySQL is one of the greatest skills for large web applications, and it is by no means a simple feat. Combine that with the frequent need for caching, and you now have a three-part solution — file system, separate database referencing, and caching — to solve the simple objective of quick and efficient file retrieval for large amounts of data. Why not start fresh with an approach that combines the need for structured access with a large file store? A few enterprising folks have done just that with an open source project called Cassandra. Billed as “distributed storage system for managing structured data while providing reliability at a massive scale,” Cassandra is basically fusing the database referencing model with the near infinite capacity of the global and clustered file systems.

Cassandra was initially developed at Facebook, where it was intended to solve the problem of Inbox search. They wanted something that was fast, reliable, and could handle significant throughput of both write and read requests. And since the messaging never stops at Facebook, they needed a system that could both store all of the data, and provide rapid, predictable results for search queries. Cassandra implements a database-style approach to managing structured data with a flexible data model than can be dynamically controlled. It bases this on a table with rows, each identified by a key, and a corresponding series of column families. A more in depth explanation can be found here.

Other potential applications beyond Inbox search include recommendation engines, targeted advertising, and content search, particularly when you combine many concurrent inputs and output requests to the same data set. Earlier this year, the project was accepted into the Apache incubator program. A project page also exists at Google Code.

Three other open-source projects are also bringing more structure back to the overall system: HBase, built on the Hadoop core; Hypertable, modeled after Google’s BigTable; and OpenNeptune. These solutions deliver rapid access to individual records within extremely large data sets — all aim (or claim) to reach petabyte scale and up to billions of rows and millions of columns. Combined, these projects represent a turn in the tides for companies relying on big data — success will be based not on how much data they can retain, but how quickly and efficiently they can use that data.

Gary Orenstein is the author of “IP Storage Networking: Straight to the Core”, host of TheCloudComputingShow.com and was a co-founder of Nishan Systems (now part of Brocade).

  1. Once you open up the “mason jar” (a file only known to its managing application), you let out the worms. At the file level we have behaved in a manner good for the OS, at the data level we have dithered over Semantics. In the database, we have added the structure at a steep cost – guaranteed scaling failures and complex brittle solutions to the same.

    Maybe this linked data pipe dream will awake from slumber, Kingsley? Where are you.

    Share
  2. [...] The storage industry has been pumping up the growth of unstructured data for years. But unstructured data without a structured means of finding, accessing, and retrieving files and objects is basically useless. Many storage administrators are now More Organizing Tips [...]

    Share
  3. [...] and includes names like CouchDB, Hypertable, HBASE, Tokyo Cabinet, LightCloud and Cassandra, which we discussed here. These projects are concisely summarized here and here. In addition, LinkedIn has made most of the [...]

    Share
  4. [...] footprint of computing. We’re taxing current technology, from the silicon to the software, with massive data demands, and technology like this is a piece of the solution, I’m [...]

    Share
  5. [...] Why Storage Needs More Structure [...]

    Share
  6. [...] a post last year, contributing writer Gary Orenstein pointed out that thanks to these attributes, Cassandra has potential applications beyond inbox search that include [...]

    Share
  7. [...] which recently emerged to provide commercial support and services for Cassandra, a popular distributed database that originated at Facebook and is now used by Twitter, Digg, [...]

    Share

Comments have been disabled for this post