Although, of all LinkBench’s features — and you can read all about them in a Facebook Engineer wall post from Monday morning — probably the biggest is that it’s open source and built to be extensible. One of the biggest problems with benchmarks overall is that they rarely align with actual production workloads inside the companies that are supposed to care about them. In this case, for example, a benchmark for measuring the performance of Facebook’s massive MySQL+memcached+Flashcache database architecture against its massive social graph and transaction activity would be all but worthless unless someone was just planning to rebuild Facebook.

I’ve written in the past that perhaps crowdsourced benchmarks are the wave of the future: essentially a compiled set of statistics and best practices as more companies test different database (or Hadoop) technologies on different hardware setups against different workloads and publish the results. Everything will of course vary by the exact details within any given environment, but it would be a good way to get a sense of how a particular stack might, or perhaps should, fare.

But an open source benchmark tuned for a specific use case — social graphs — by probably the world’s foremost expert on that use case is interesting, too. Anyone else trying to serve data from their own social graphs can benefit from some of LinkBench’s more-prominent features, such as its ability to generate “large synthetic social graphs,” while tuning it to the specifics of their own infrastructure. After all, it might be that your app has different requirement around reading versus writing data, and it’s very possible you’re not using MySQL, either.

Or maybe you are using MySQL and want to see how a newer database technology might handle your graph workload. That, by the way, is one of the reasons Facebook built LinkBench, according to this post.

At any rate, the social web is all about graphs, and database performance really matters for anyone trying to build a service that stays online and provides a pleasant user experience. Say what you want about Facebook, but its services perform, so the bar is set high for anyone trying to dethrone it or at least to build something than can attract an equally large and devout following.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Is the Future of Enterprise Completely Open Source?
• How devops can reduce cycle times
• Migrating media applications to the private cloud: best practices for businesses

The researchers — who come from the University of Southern California, Scripps Clinic, the Scripps Research Institute, the University of California, San Diego Moores Cancer Center and Memorial Sloan-Kettering — combined autopsy data from 163 cancer cases (all from before the advent of radiation therapy in order to analyze the natural spread) with applied mathematics in order to carry out their study. What they found, according to a press release about the research is that

metastatic lung cancer does not progress in a single direction from primary tumor site to distant locations, which has been the traditional medical view. Instead … cancer cell movement around the body likely occurs in more than one direction at a time.

How cancer cells spread. Source: PLOS One

Moreover, they found certain organs tend to spread cancer cells more aggressively, while others tend to act as sponges for cancer cells. These sponge organs might still grow tumors, they just don’t disperse the cells.

The PageRank analogy

The mathematics involved here — called Markov chain models — are similar to what Google uses to determine what web pages are the highest-quality for any given search query. Only whereas Google uses the number and quality of links to determine the probability of a web surfer landing on any given page, these researchers are trying to predict the PageRank of tumors, if you will. So, generally speaking, a kidney would likely have a higher PageRank than a liver because the kidney is more likely to spread cancer cells throughout the body (or, in web-search terms, generate a lot of links to itself).

The network path of cancer cells from lung to liver. Source: PLOS One

As data volumes proliferate and relationships between data points become more complex, Markov models are actually becoming pretty popular. Netflix uses them in order to predict the movies users will want to watch next.

The weighted connections between various states or web pages or whatever someone is ranking are often expressed as the nodes and edges of a graph. Graphs, of course, have become part of the everyday web lexicon thanks to the various social graphs and interest graphs that analyze who we’re connected to (and how) and the types of topics we browse online.

The web as a data science proving ground

So in the end, perhaps, the most-important contribution of the worldwide web won’t be the revolution in terms of how we access information, but the web’s function as a proving ground for advanced statistical methods starring very large and complex data sets like those found in the medical world. Already, for example, another group of medical researchers has used a Markov variant in order to create a model they think can prescribe better treatment plans because it analyzes the costs and patient outcomes usually associated with a given treatment for a given symptom.

Tracking a cholera outbreak across a river network. Source: Physical Review Letters

Last year, a group of Swiss researchers developed an algorithm that, having access to a relatively small amount of data, can track anything from Twitter rumors to disease outbreaks back to their source. A company called Syapse uses the graph structure to chart the relationships among words across different medical specialties.

One would also be remiss in ignoring the computing and data-storage innovation spurred by the web that has improved our ability to handle massive amounts of genetic and other data. As the lung cancer researchers explain in their paper:

One of the strengths of such a statistical approach is that we need not offer specific biomechanical, genetic, or biochemical reasons for the spread from one site to another, those reasons presumably will become available through more research on the interactions between CTCs and their microenvironment. We [have created] a quantitative and computational framework for the seed-and-soil hypothesis as an ensemble based first step, [that] then can be further refined primarily by using larger, better, and more targeted databases such as ones that focus on specific genotypes or phenotypes, or by more refined modeling of the correlations between the trapping of a CTC at a specific site, and the probability of secondary tumor growth at that location.

The long story short is that the more data we have and the easier we can analyze and map it, the better we can treat — and perhaps even cure — cancer and other complicated diseases.

Feature image is a network map of how lung cancer spreads between organs, where each numbered node correlates with a specific organ.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Connected world: the consumer technology revolution
• The importance of putting the U and I in visualization

Of course, this being a Reddit discussion, Rasmussen answers a bunch of questions about the history of graph search, its privacy issues, his role with building Google Maps (and Wave) and his walks with Mark Zuckerberg. But here are some of the more-informative excerpts about the architecture itself.

I should point out, too, that we’ll be talking a bit about graph processing and graph databases at our Structure: Data conference next month, too. Graphs, as it turns out, are a great way to storing, processing and presenting a lot of data that has nothing to do with social connections.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 10 ways big data changes everything
• 9 Companies that Pushed the Infrastructure Discussion in 2010
• How search can unlock the power of big data

But to understand how Syapse works, you have to understand the problem it’s trying to solve. A condensed version of the situation is this: Sequencing genomes, proteomes, biomes and other microscopic, but very important, biological players generates a lot of data. However, we’re not just talking about the terabytes of data that a fully sequenced genome (or perhaps the tens of thousands sequenced gut bacteria, which can change composition hourly) will produce, but also patient data (e.g., name, date of birth, smoker or non-smoker, etc.) and process data (i.e., everything that happens from the time a lab gets a sample to the time a doctor gets a report on his desk).

The complexity and perpetually changing nature of both the field of “omics” as it’s called, and the data itself, further complicates things. According to Syapse Co-founder and President Jonathan Hirsch, diagnostics labs and workers are always using new and different processes trying to optimally extract, tag and analyze samples. Furthermore, expert knowledge of what any particular genetic or other signature means is always changing (for example, Hirsch said, we only really understand about 1 percent of the human genome), as are the ontologies that lab workers, researchers and physician specialists use as their particular fields evolve.

“There is basically a wholes set of measurements that go beyond just sequencing the genome,” he explained. Analyzing genomes, proteomes and anything else is “like a very, very complicated recipe” that involves much more than swabbing someone’s cheek and getting back a comprehensive, understandable report. Syapse doesn’t actually do any of the sequencing work (like a DNAnexus or Bina Technologies does,) but just captures the metadata from those lab processes and connects to those hefty sequenced data via an API so the platform has access to everything it needs.

Organizing complex data requires a graph

Using semantic-analysis and graph-processing techniques, Syapse thinks it can bring the world of “omics” under control. Although it’s currently working with research centers that analyze the data in order to better hone their processes, Hirsch expects the company will eventually make most of its money from doctors and hospitals using Syapse to help better diagnose their patients. “[We're] trying to fill the gap and be the company that cracks the physician side of this,” he said.

This is where the Graph Search comparison comes into play. The Syapse platform is continuously updated with the latest ontologies from various fields and the changing meanings of the metadata associated with the various lab processes. All this information is stored based on its relationship to other pieces, and semantic analysis means the Syapse software knows that Term X in one field might actually mean Term Y in another.

Syapse has essentially created a “huge knowledge graph” of clinical, diagnosis and omics data, Hirsch explained, and doctors and researchers can mine it using whatever terms they use in their daily lives. They can easily search, for example, by patients they’ve treated for breast cancer whose genes showed certain specific markers and were processed using particular techniques in the lab in order to find connections among them.

Syapse Co-founder and CEO Glenn Winokur — an admitted “IT guy” compared with his biotech-focused partners — likes to put the platform’s promise in the terms of business software. ”Think of this entire workflow as similar to a sales or marketing workflow,” he said, adding that Syapse is trying to make mining omics data as simple for its users as Salesforce.com makes CRM for its users.

That’s probably a good analogy for selling the software to hospital administrators who might be more concerned with budgets than with big data technology. As we’ll discuss in more detail at our Structure: Data conference on March 20-21, business people are increasingly concerned with using data to make better decisions, but they need applications that make it easier and faster to find stuff out than is possible with many open source packages targeting engineers and statisticians. If Syapse can deliver on this promise for making sense of our complex biological systems, it could make a big difference.

Feature image courtesy of Shutterstock user kentoh; lab image courtesy of Shutterstock user VILevi.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• AWS Storage Gateway jolts cloud-storage ecosystem
• 9 Companies that Pushed the Infrastructure Discussion in 2010

As I explained in March when, Gravity has built a semantic-analysis engine that tries to gauge a site visitor’s interest by looking at more than the articles that person reads. Thanks to an expansive database of topics and a hybrid man-machine machine learning system that takes into account behavior as well as content, Gravity can determine other topics that might be of interest even if those connections aren’t visible to the naked eye. The result of this analysis is called an interest graph, which is like a social graph only that’s concerned with interests rather than people.

Currently, Gravity claims its total body of graph data exceeds 18 million megabytes, or 18 terabytes. The company says the new money will help it expand operations in the United States and even deploy its own content-marketing platform.

Of course, interest graphs are useful for more than just automatically presenting visitors with the news content they’re interested in. Gravity also has a product for advertisers to better target potential customers, and an analytics service so publishers can get in-depth visualizations of who’s reading their content and what content works better than other content.

However, the obvious elephant in the room when talking about interest graphs is privacy and how to collect and analyze user data without crossing any ethical guidelines. This will become even more of an issue as web platforms try to share data across services in order to create a more unified browsing experience in which interest graphs follow users around the web to inform personalization algorithms at every step. And as we’ll discuss later this month at our Structure: Europe conference in Amsterdam, some governments take user privacy much more seriously than others, which can make businesses based on that data a little trickier to operate.

Here’s Gravity Co-Founder and CTO Jim Benedetto, along with privacy attorney Ashlie Beringer, discussing the issue with me at our Structure: Data conference last March.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• Connected world: the consumer technology revolution

It’s therefore far from controversial to say that Hadoop is firmly planted in the enterprise as the big data standard and will likely remain firmly entrenched for at least another decade. But, building on some previous discussion, I’m going to go out on a limb and ask, “Is the enterprise buying into a technology whose best day has already passed?”

First, there were Google File System and Google MapReduce

To study this question we need to return to Hadoop’s inspiration – Google’s MapReduce. Confronted with a data explosion, Google engineers Jeff Dean and Sanjay Ghemawat architected (and published!) two seminal systems: the Google File System (GFS) and Google MapReduce (GMR). The former was a brilliantly pragmatic solution to exabyte-scale data management using commodity hardware. The latter was an equally brilliant implementation of a long-standing design pattern applied to massively parallel processing of said data on said commodity machines.

GMR’s brilliance was to make big data processing approachable to Google’s typical user/developer and to make it fast and fault tolerant. Simply put, it boiled data processing at scale down to the bare essentials and took care of everything else. GFS and GMR became the core of the processing engine used to crawl, analyze, and rank web pages into the giant inverted index that we all use daily at google.com. This was clearly a major advantage for Google.

Enter reverse engineering in the open source world, and, voila, Apache Hadoop — comprised of the Hadoop Distributed File System and Hadoop MapReduce — was born in the image of GFS and GMR. Yes, Hadoop is developing into an ecosystem of projects that touch nearly all parts of data management and processing. But, at its core, it is a MapReduce system. Your code is turned into map and reduce jobs, and Hadoop runs those jobs for you.

Then Google evolved. Can Hadoop catch up?

Most interesting to me, however, is that GMR no longer holds such prominence in the Google stack. Just as the enterprise is locking into MapReduce, Google seems to be moving past it. In fact, many of the technologies I’m going to discuss below aren’t even new; they date back the second half of the last decade, mere years after the seminal GMR paper was in print.

Here are technologies that I hope will ultimately seed the post-Hadoop era. While many Apache projects and commercial Hadoop distributions are actively trying to address some of the issues below via technologies and features such as HBase, Hive and Next-Generation MapReduce (aka YARN), it is my opinion that it will require new, non-MapReduce-based architectures that leverage the Hadoop core (HDFS and Zookeeper) to truly compete with Google’s technology. (A more technical exposition with published benchmarks is available at http://www.slideshare.net/mlmilleratmit/gluecon-miller-horizon.)

Percolator for incremental indexing and analysis of frequently changing datasets. Hadoop is a big machine. Once you get it up to speed it’s great at crunching your data. Get the disks spinning forward as fast as you can. However, each time you want to analyze the data (say after adding, modifying or deleting data) you have to stream over the entire dataset. If your dataset is always growing, this means your analysis time also grows without bound.

So, how does Google manage to make its search results increasingly real-time? By displacing GMR in favor of an incremental processing engine called Percolator. By dealing only with new, modified, or deleted documents and using secondary indices to efficiently catalog and query the resulting output, Google was able to dramatically decrease the time to value. As the authors of the Percolator paper write, ”[C]onverting the indexing system to an incremental system … reduced the average document processing latency by a factor of 100.” This means that new content on the Web could be indexed 100 times faster than possible using the MapReduce system!

Coming from the Large Hadron Collider (an ever-growing big data corpus), this topic is near and dear to my heart. Some datasets simply never stop growing. It is why we baked a similar approach deep into the Cloudant data layer service, it is why trigger-based processing is now available in HBase, and it is a primary reason that Twitter Storm is gaining momentum for real-time processing of stream data.

Dremel for ad hoc analytics. Google and the Hadoop ecosystem worked very hard to make MapReduce an approachable tool for ad hoc analyses. From Sawzall through Pig and Hive, many interface layers have been built. Yet, for all of the SQL-like familiarity, they ignore one fundamental reality – MapReduce (and thereby Hadoop) is purpose-built for organized data processing (jobs). It is baked from the core for workflows, not ad hoc exploration.

In stark contrast, many BI/analytics queries are fundamentally ad hoc, interactive, low-latency analyses. Not only is writing map and reduce workflows prohibitive for many analysts, but waiting minutes for jobs to start and hours for workflows to complete is not conducive to the interactive experience. Therefore, Google invented Dremel (now exposed as the BigQuery product) as a purpose-built tool to allow analysts to scan over petabytes of data in seconds to answer ad hoc queries and, presumably, power compelling visualizations.

Google BigQuery

Google’s Dremel paper says it is “capable of running aggregation queries over trillions of rows in seconds,” and the same paper notes that running identical queries in standard MapReduce is approximately 100 times slower than in Dremel. Most impressive, however, is real world data from production systems at Google, where the vast majority of Dremel queries complete in less than 10 seconds, a time well below the typical latencies of even beginning execution of a MapReduce workflow and its associated jobs.

Interestingly, I’m not aware of any compelling open source alternatives to Dremel at the time of this writing and consider this a fantastic BI/analytics opportunity.

Pregel for analyzing graph data. Google MapReduce was purpose-built for crawling and analyzing the world’s largest graph data structure – the internet. However, certain core assumptions of MapReduce are at fundamental odds with analyzing networks of people, telecommunications equipment, documents and other graph data structures. For example, calculation of the single-source shortest path (SSSP) through a graph requires copying the graph forward to future MapReduce passes, an amazingly inefficient approach and simply untenable at scale.

Therefore, Google built Pregel, a large bulk synchronous processing application for petabyte -scale graph processing on distributed commodity machines. The results are impressive. In contrast to Hadoop, which often causes exponential data amplification in graph processing, Pregel is able to naturally and efficiently execute graph algorithms such as SSSP or PageRank in dramatically shorter time and with significantly less complicated code. Most stunning is the published data demonstrating processing on billions of nodes with trillions of edges in mere minutes, with a near linear scaling of execution time with graph size.

At the time of writing, the only viable option in the open source world is Giraph, an early Apache incubator project that leverages HDFS and Zookeeper. There’s another project called Golden Orb available on GitHub.

In summary, Hadoop is an incredible tool for large-scale data processing on clusters of commodity hardware. But if you’re trying to process dynamic data sets, ad-hoc analytics or graph data structures, Google’s own actions clearly demonstrate better alternatives to the MapReduce paradigm. Percolator, Dremel and Pregel make an impressive trio and comprise the new canon of big data. I would be shocked if they don’t have a similar impact on IT as Google’s original big three of GFS, GMR, and BigTable have had.

Mike Miller (@mlmilleratmit) is chief scientist and co-founder at Cloudant, and Affiliate Professor of Particle Physics at University of Washington.

Feature image courtesy of Shutterstock user Jason Prince; evolution of the wheel image courtesy of Shutterstock user James Steidl.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The importance of putting the U and I in visualization
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Technically, the “NoSQL” label applies to a broad collection of databases that do a wide variety of things, with a single unifying characteristic: that they’re not SQL. Although document-oriented systems (such as MongoDB and CouchDB) and key-value stores (such as Cassandra) get much of the attention because they address common issues around scale, speed and flexibility for web companies and even large enterprises, graph databases are finding their own voice.

As I explained previously while covering GoldenOrb, another graph database:

Essentially, graph databases excel at finding relationships between disparate pieces of data, with one major use case being social graphs. They run analyses over terabytes of graph data while maintaining the relationships between the data, even as the data and the relationships constantly evolve.

Aside from GoldenOrb, an open source product from Austin, Texas-based startup Ravel Data, we’ve also recently seen database veteran Objectivity work its way into the NoSQL space with its InfiniteGraph product.

Graph databases have actually been around for a while, but they’ve gained some popularity recently because of their use with large web properties such as Google and Twitter. Google built its own called Pregel, and Twitter built one called FlockDB.

For its part, Neo4j is an open source database for which the Menlo Park, Calif.-based Neo Technology offers support, services and commercial licenses. Fidelity Growth Partners led this funding round for Neo — its first — along with seed investors Sunstone Capital and Conor Venture Partners.

Images courtesy of Neo4j.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Essentially, graph databases excel at finding relationships between disparate pieces of data, with one major use case being social graphs. They run analyses over terabytes of graph data while maintaining the relationships between the data, even as the data and the relationships constantly evolve.

Twitter actually created its own graph database, called FlockDB, to help the site determine who’s connected to whom in the Twittersphere. Google uses Pregel to power its PageRank feature, although as it explained in a 2009 blog post introducing the technology, there are many other possibilities:

If you squint the right way, you will notice that graphs are everywhere. For example, social networks, popularized by Web 2.0, are graphs that describe relationships among people. Transportation routes create a graph of physical connections among geographical locations. Paths of disease outbreaks form a graph, as do games among soccer teams, computer network topologies, and citations among scientific papers. Perhaps the most pervasive graph is the web itself, where documents are vertices and links are edges. …

A relatively simple analysis of a standard map (a graph!) can provide the shortest route between two cities. But progressively more sophisticated analysis could be applied to richer information such as speed limits, expected traffic jams, roadworks and even weather conditions. In addition to the shortest route, measured as sheer distance, you could learn about the most scenic route, or the most fuel-efficient one, or the one which has the most rest areas. All these options, and more, can all be extracted from the graph and made useful — provided you have the right tools and inputs.

In spreading the word about GoldenOrb, Ravel expands upon the use cases, citing marketing analysis, pharmaceutical research and, essentially, any situation in which it would be beneficial to “run traditional analytics on entire data sets instead of only small samples … .”

A couple of things make GoldenOrb particularly worth watching: 1) it’s both an open source and a product, which distinguishes it from Twitter’s open-source FlockDB project, Google’s proprietary Pregel project and Objectivity’s proprietary InfiniteGraph product; and 2) it’s based on Hadoop. Having an actual product to work on instead of just code could garner a large community, especially from the growing ranks of Hadoop developers.

Hadoop and NoSQL databases have both ridden the big data wave to form robust development communities, so why can’t graph databases be next? The GoldenOrb code is available at https://github.com/raveldata/goldenorb.

For more information about Ravel from the horse’s mouth, including plans to create an enterprise version of Apache’s Hadoop-based Mahout machine-learning platform, check out this video interview with Ravel president Zach Richardson:

Image courtesy of Ravel.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Defining Hadoop: the Players, Technologies and Challenges of 2011