Can LexisNexis build a Hadoop-killer?

Mathew Ingram — Thu, 22 Mar 2012 15:47:37 +0000

LexisNexis is a giant company, part of the Reed Elsevier information empire, but when it comes to handling big data, it is in the unusual position of being the underdog. The leader of the pack is Hadoop, which has already amassed a large and rapidly-growing following for its ability to manage large databases — but Armando Escalante of LexisNexis told attendees at GigaOM’s Structure:Data conference in New York on Thursday that he believes the company has built what ~~could be~~ others might call a Hadoop killer. Originally developed to handle LexisNexis’ own internal data needs, the HPCC system was open-sourced nine months ago, and Escalante said it is already outperforming Hadoop in a number of ways.

Because LexisNexis has so much data that it needs to analyze and provide to clients for its legal and government services, Escalante said that the company began building its own internal data-handling platform almost a decade ago, before “big data” even became a buzzword. “We already run our business on this, end-to-end,” he said. Once it became obvious that Hadoop was becoming a popular solution, LexisNexis decided to open-source the project and use the knowledge of a community of users and developers to improve and expand it.

Escalante said the LexisNexis’ system offers a number of features that Hadoop doesn’t, including a big-data delivery engine, and that it is building a layer that will allow its system to handle data from Hadoop. In fact, he said in a recent test a single LexisNexis node was 20-percent faster than a multi-node Hadoop configuration. But the biggest advantage that LexisNexis has, according to Escalante, is that because it is a large company and has already been using the system internally for years, the banks and insurance companies that make up a majority of its clients are more likely to want to use it than Hadoop.

“We have most of the banks and insurance companies as clients, and we are doing proof-of-concept tests with many of them now, and I think they may be more comfortable working with a company that’s not a startup,” Escalante said. “Big companies want a neck to squeeze sometimes, and LexisNexis has a big neck.”

Watch the livestream of Structure:Data here.

Update: This post was updated to reflect that moderator Derrick Harris described the project as a “Hadoop killer,” not Escalante.

Watch live streaming video from gigaombigdata at livestream.com

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

GigaOM » Lexis-Nexis

Can LexisNexis build a Hadoop-killer?