Graph analysis is among the hottest techniques around for making sense of large datasets, primarily by determining how tightly different data points are related or how similar they are. The term “graph” came into the broader lexicon along with social networks, which built social graphs to assess the relationships among their millions of users, but the technique has much broader uses.

My LinkedIn social graph

Guestrin said GraphLab’s algorithms are used in a lot of recommender systems, but he also cites fraud detection in banking networks and intrusion detection in computer networks as potential applications. We’ve covered graphs as the analytical model of choice for everything from content recommendation to tracking lab work in genomics. Really, though — especially when combined with machine learning — graph analysis can be applied to anything where there’s too much data for a person to possibly analyze the relationships between every point.

One of Ayasdi’s graph-like data maps

Google also famously uses a graph-processing system called Pregel as part of PageRank. Although a number of graph databases and other projects have popped up in the past few years, Guestrin said GraphLab is actually a contemporary of Pregel. He and some colleagues at Carnegie Mellon built a small system for their lab about five years ago, then released it into the open-source world with few expectations that it would catch on. Now, he added, Pandora and WalmartLabs are among the project’s user base.

Among those other projects are graph databases such as Giraph (an open source, Hadoop-based Pregel clone developed at Facebook) and Neo4j (which also has a commercial arm, called Neo Technology), as well as Twitter’s Cassovary and fellow University of Washington project Grappa. Guestrin said GraphLab can work with most of them, particularly if they’re not designed to do machine learning at scale like GraphLab is. Some efforts, he noted, are focused on simply storing data in graph form (e.g., databases) or in providing simple graph analysis.

As for when we’ll actually see the results of the effort to commercialize GraphLab, Guestrin said it will be a while. Right now, he’s focused on the next open source release of GraphLab in July. However, the company will begin engaging with commercial users over the next several months to determine what types of features they would expect in commercial graph-analysis software.

The bigger question to come out of all this graph activity, though, is how big a market we’ll ultimately see for graph-analysis or any other specific technique. As companies get more comfortable with big data from a technical standpoint, they’re getting more interested in the different types of analysis it allows for too. This is evidenced by the quest to make Hadoop support myriad processing frameworks aside from MapReduce.

We already have a handful of commercial graph products on the market — including an industrial grade one called YarcData from supercomputer maker Cray — but how many will there eventually be? And if graph analysis is all the rage right now, what comes next?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Connected world: the consumer technology revolution

Gravity’s recommendation system is based on its interest graph technology, which we detailed last year. Here’s how I described it then:

[T]he gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

As new content streams into Gravity’s system, it’s analyzed and categorized in real time, then presented to users accordingly based on their interests and behavioral history.

How Gravity’s platform works

Graph processing and graph databases — which store and analyze data based on their relationship to one another — are critical to our onlines lives, powering everything from online recommendations to social search to knowledge discovery. Graph technologies are also the focal point of some impressive life sciences work from companies such as Syapse and Ayasdi, which will be presenting at Structure: Data in New York next month.

But publishers struggling to stand out on a noisy web might have the most to gain from graphs and personalization, generally. At our PaidContent Live conference (April 17 in New York), executives from Prismatic, Zite and Bluefin Labs will take the stage to talk about the importance of personalization for helping consumers filter through the deluge of content online so they can find what they really want. It’s arguable that the trick to keeping readers happy is knowing what they want to read — possibly better than they do themselves.

According to Gravity, its platform currently “delivers more than 25 million personalized content recommendations per day to more than 200 million users. Beta partners have reported click through rates two to three times above previous levels, return visitation increases of 300 percent and session length increases up to 40 percent.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization

We’ve covered Neo and its technology, a commercial version of the open source Neo4j graph database, before. Graph databases store data based on their relationship to one another, making them ideal for organizing massive amounts of information such as social networks, the connections between various web pages and web data, or, in the case of Neo Technology customer Cisco, its master data management system. Neo4j, Neo Founder and CEO Emil Eifrem told me last year, is also ideal for serving transactions and as part of a Java application stack, which helps its explain why VMware’s SpringSource division is backing the project.

Neo’s customer base also includes Deutsche Telekom, Lockheed Martin, BAE Systems, LexisNexis, Adobe and Pitney Bowes, as well as a slew of web startups.

Aside from graph databases, those focused on storing data relating to time and place are also catching fire. In August, spacial-temporal database startup SpaceCurve raised a $3.5 million second round for its technology that the company claims can make short work of uncovering patterns in complex geospatial and time-series data. TempoDB, a startup offering a cloud database service for time-series data like that thrown off of sensors and electrical equipment, has raised almost $900,000 in seed funding. Space-Time Insights, a company that specializes in visualizing organizations’ geospatial and time-series data just raised $14 million.

This, of course, is only a sampling of the types of databases that are garnering attention as we generate more data from more sources and need new ways of storing, accessing and analyzing that data. The relational database isn’t going away any time soon, but it looks like it might soon be sitting alongside a variety of databases designed for other tasks within a lot of corporate data centers.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Emerging trends in the non-relational database market
• AWS Storage Gateway jolts cloud-storage ecosystem
• A near-term outlook for big data

Aspiring heirs to the Klout throne, for example, might look to a project called STINGER currently under development at Georgia Tech University. STINGER, which stands for Spatio-Temporal Interaction Networks and Graphs Extensible Representation, is a graph-processing engine that project lead David Bader says is bigger, faster and more flexible than anything currently in use for analyzing social media connections. You provide a shared-memory computing system, and it provides an open-source tool that can help detect relationships between billions of people, places and things as those relationships change over time — even in real time.

Someone using Facebook data, for example, might write an algorithm using where people or pages would be the vertices and actions (likes, shares, wall posts, etc.) would be the graph’s edges. One relatively easy application, Bader explained, would be to analyze how activity around particular people is increasing, decreasing or changing, therefore indicating changes in their importance or the growth of new communities.

We’ll do the hard work

Writing an algorithm to perform that kind of analysis isn’t really the problem, though — it’s writing one that can scale into the billions of vertices and edges and still perform quickly enough to be useful. An algorithm that generates one false positive in a million isn’t so bad when you’re dealing with tens of thousands of items, Bader explained, but it gets to be a big problem when you’re talking about billions of items against which it’s running.

There are dozens of open source graph databases available, including popular offerings such as Neo4j and InfiniteGraph, but he said, “Our lab focuses on algorithms that run fast on massive data sets and that are more accurate than what is traditionally done in social media.”

David Bader

Bader’s team recently presented a paper detailing a social media algorithm running atop STINGER that ran 100 times faster than some previous approaches because the system stores the graph’s previous state and only performs the minimal amount of processing necessary as new edges are inserted. This is in contrast to traditional approaches that re-process the entire graph every time there’s a change.

That being said, Georgia Tech isn’t entirely alone analyzing massive amounts of social data with graph databases. Google’s Pregel had already scaled to billions of vertices and edges as of 2009, and Facebook is currently analyzing more than a billion edges using Apache Giraph (an open source, Hadoop-based Pregel implementation). But those cases — both companies are loaded with smart engineers, data scientists and powerful infrastructure — just underscore the importance of what researchers like Bader are building and releasing as open source.

Forget social media, solve real problems

But social data isn’t just useful for figuring out who’s influential on Twitter or Facebook — it also can be used to solve some real problems. Bader said he’s already used graph processing with Twitter data to determine who was leading resistance units during Egypt’s recent revolution. ”Anywhere I can look at connections between entities,” he said, “these approaches are available.”

Indeed. On Wednesday afternoon, for example, a group of researchers from the University of Alberta, University of Connecticut and University of California-Merced unveiled a new data-based method that could make it faster, easier and less expensive to root out culprits in fraud cases.

The technique uses a method called the Steiner tree to analyze the connections –social, business, familial, etc. — between the people involved in a given case of fraud. The algorithm is able to determine the shortest path between two objects, which the researchers posit is especially applicable to fraud investigations — the person with the shortest path between himself and the crime is probably the culprit (or at least a solid suspect).

The fraud researchers’ paper follows the publication in August of a method for determining the source of everything from a disease outbreak to a Twitter rumor by tracking its spread across a complex network over time. Their algorithm, the paper’s authors claim, could be particularly effective for combating cybercrime by tracking computer viruses back to their sources. The more connections (in the case of social data), or observers, a particular point has, the fewer that are needed to track down the source point.

However, all the algorithms and data frameworks in the world probably won’t make too big a difference until they’re turned into products that actually work on real-world situations. As the University of Alberta’s Ray Patterson pointed out in a press release detailing the fraudster-detection algorithm, ”It might take several years or many years before anyone picks it up. But it’s a good thing if we can point people towards what’s useful.”

Georgia Tech’s Bader said DARPA, Intel, Sandia National Laboratory and other research institutions have already used STINGER to tackle some complex data sets, and he suspects a strong commercial interest, as well. If a company is willing to take STINGER from a project into a product, it could bring the project’s scale and speed to everything from analyzing customer interactions to monitoring the changing nature of criminal networks, Bader said. Considering the desire from companies of all types to extract some meaning from social data, I have to think someone will give it a shot.

Feature image courtesy of Shutterstock user 3DProfi.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• 12 tech leaders’ resolutions for 2012
• NewNet Q4: Platform mania and social commerce shakeout

For Google, it’s all about collecting and analyzing billions of data points to learn what each one really means. With Knowledge Graph, for example, Google uses a “database of more than 500 million real-world people, places and things with 3.5 billion attributes and connections among them.” It’s those connections that are the key, as they’re what make the system smart enough to know what you’re looking for that wouldn’t naturally show up in a standard keyword search.

Although Google hasn’t come out and said so, I’d imagine the Knowledge Graph utilizes Google’s Pregel graph processing engine. Graph processing and databases are catching on in social networks and other large-scale environments because they organize pieces of data by how they’re connected to one another. Those connections are called edges, and they’d keep Knowledge Graph results both informative and focused because the system knows how closely they’re related in any given circumstance.

This example of a personalized interest graph from Gravity Labs illustrates how one might visualize a graph, in this case the connections between a reader’s perceived interests:

Of course, Google has another tool at its disposal, which is the collective wisdom it’s able to glean from billions of searches every day. So, as Singhal wrote when first explaining Knowledge Graph in May, “[W]e can now sometimes help answer your next question before you’ve asked it, because the facts we show are informed by what other people have searched for. For example, the information we show for Tom Cruise answers 37 percent of next queries that people ask about him.”

Google’s other big announcement today is improved voice search on mobile phones, both Android and iOS. Here’s how Singhal describes the new capability:

You just need to tap the microphone icon and ask your question, the same way you’d ask a friend. For example, ask “What movies are playing this weekend?” and you’ll see your words streamed back to you quickly as you speak. Then Google will show you a list of the latest movies in theaters near you, with schedules and even trailers. … When Google can supply a direct answer to your question, you’ll get a spoken response too.

On Monday, a Google Research blog post noted how the company’s work on neural networks — which it famously used to train a system capable of detecting cats and human faces in video streams — is being used to power speech recognition in the Jelly Bean release of Android. Seventeen-year-old Brittany Wenger recently won the Google Science Fair by building an application atop Google App Engine that uses a neural network to help detect breast cancer.

As one might imagine, however, the big challenge for Google, Microsoft , Apple and everyone else trying to provide intelligent but intuitive user experiences is figuring out how to shape high computer science into easily digestible formats on ever-smaller devices. Search would certainly be a more effective tool if everyone could write complex queries directly against a company’s database, but the trick is making products good enough that we don’t have to. It’s boiling years of machine learning, natural-language processing and neural network research into “you ask a question and your phone spits back the right answer.”

Feature image courtesy of Shutterstock user Sebastian Kaulitzki.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Further, SpringSource founder and GM (and current SVP of applications platforms at VMware) Rod Johnson is now chairman of Neo’s board of directors. VMware has been nothing if not strategic in how it has grown the SpringSource business since acquiring it in 2009, so the alignment with Neo could foreshadow a stronger NoSQL push by VMware.

What’s particularly interesting is that Emil Eifrem, Neo Technology CEO and one of Neo4j’s creators, told me Spring Data Neo4j is the product of a joint development between himself and SpringSource’s Johnson. And not only is Johnson now chairman of Neo’s board of directors, but as far as Eifrem knows, it’s Johnson’s only board seat of any kind.

The seemingly tight connection doesn’t mean VMware will start pushing Neo4j as the only NoSQL database that matters, or buy Neo Technology tomorrow — Spring Data Neo4j is actually part of the greater Spring Data project that includes a number of other NoSQL options — but it does underscore VMware’s ongoing commitment to NoSQL and other alternatives to legacy relational databases. Last year, it purchased distributed data grid vendor GemStone Systems and hired Redis key developer Salvatore Sanfilippo.

As Eifrem said, both Oracle and MySQL established their lofty positions by being the right tool for the application evolutions that were happening around the time of their respective births. MySQL was the “M” in the LAMP stack that underpins so many first-generation web applications, he said, so “what is the LAMP stack of the cloud?” Perhaps it’s a NoSQL database like Neo4j.

Graph databases such as Neo4j are particularly well suited for social media uses because of their ability to draw connections between different pieces of data, but Eifrem said Cisco already is using Neo4j for a large master data management system. And, although not wholly uncommon among NoSQL databases, Eifrem said Neo4j also provides support for both transactions and Java environments, which makes it a nice fit within the Spring developer community comprised of approximately 3 million enterprise developers working programming largely in Java.

The Menlo Park, Calif.-based Neo closed a $10.6 million Series B round in September.

Feature image courtesy of Flickr user opticalreflex. Graph database model courtesy of Neo Technology.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Emerging trends in the non-relational database market
• Infrastructure Overview, Q2 2010
• What VMware’s SpringSource Acquisition Means for Microsoft