Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Last week, I explained how HBase is fast becoming the star of the Hadoop ecosystem because it allows users to build more real-time, almost transactional applications on top of Hadoop. True to its form with its other products, MapR has taken HBase even further with M7 by promising greater availability (99.999 percent), instant recovery, faster operations and the ability to handle 1 trillion tables in a single cluster. In open source versions of HBase, MapR VP of Marketing Jack Norris told me, the accepted table limit per cluster is several hundred.

Additionally, M7 shares a single data layer with the Hadoop file system, meaning less performance overhead and, presumably, easier management.

As we’re seeing with other Hadoop vendors, including Cloudera (which released its Impala SQL query engine on Tuesday), the Hadoop market is fast becoming one where each vendor is trying to set itself apart from the rest by building the best platform with the broadest set of capabilities. In furtherance of that mission, MapR also announced on Wednesday full-text search on its Hadoop distribution thanks to a partnership with Lucene specialist LucidWorks. It already has its own Hadoop distribution complete with proprietary code to bolster the file system and speed up MapReduce, as well as an open source SQL-on-Hadoop project called Drill in the works.

MapR employees are probably sleeping a lot easier these days as a result of this platform push. Others in the Hadoop market used to talk about the fear of fragmentation and then point at MapR as the example of a company helping foment that outcome with its proprietary software. Now, however, even if everyone else is building open source products, they’re all still backing their own and largely dismissing the others.

I suspect the result is feature lock-in even there’s no technological lock-in, kind of like using Amazon Web Services for cloud computing and then hoping to replicate its various servies elsewhere. It might be easy enough to move your data, but impossible or very difficult to replicate those additional capabilities elsewhere. If MapR can build a better version of HBase and companies are willing to pay for it, then so be it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Big data 2013: key trends and companies to watch

Launched as a private beta in May 2012 and made public in October, Impala is Cloudera’s attempt to address the growing demand for interactive SQL analytics on Hadoop data. It’s essentially a massively parallel database designed to share the same storage platform and metadata as Hadoop MapReduce, only it is its own separate processing engine.

How Impala fits in

Impala actually uses the same “nearly ANSI” version of SQL as does current standard bearer Hive, but that technology (created by Facebook in 2009 as a data warehouse layer for Hadoop) doesn’t run nearly fast enough to sate many users’ desire for interactive analytics. This is because Hive transforms SQL queries into MapReduce jobs, meaning every one is processed against the entire corpus of data in the Hadoop Distributed File System.

Sizing up the competition

Only Cloudera isn’t the first to have the idea, nor is it alone in trying to sell interactive SQL on Hadoop. The idea was first commercialized by Boston-based startup Hadapt in 2011, and is now being pushed by numerous startups and larger Hadoop players. Among them: Pivotal (formerly EMC) Greenplum, MapR (with Drill), Hortonworks (with Stinger), Drawn to Scale, Splice Machine, Jethro Data and Citus Data.

Hadapt’s architecture

But Cloudera is arguably the biggest name pushing SQL on Hadoop, and CEO Mike Olson thinks Impala stands out for several reasons — not the least of which is that it exists as a product. “Nobody else is shipping production-grade SQL query support on Hadoop,” he told me during a recent call. “At least not in open source.” He seems content to let the startups do their things, instead focusing his attention on Cloudera’s big three Hadoop-distribution competitors in Pivotal, MapR and Hortonworks. Greenplum and Pivotal SVP Scott Yara was full of confidence — and R&D budget– when the company announced the Pivotal HD distribution and HAWQ technology in February, but Olson claims the approach requires a siloed DBMS within HDFS and is a “rearguard defensive strategy” to protect the company’s sunk costs in its database technology.

The Pivotal HD and Hawq architecture

As for Hortonworks, Olson questions the wisdom of its Stinger initiative to boost Hive’s speed, noting that “Hive never got good while it was running standalone on MapReduce.” Hortonworks also partners with vendors such as Teradata to let their platforms access Hadoop data in its native format, but those approaches still require sending data over the network. “It’s not the way you would build it if you woke up in the 2000s and were building this anew,” Olson said.

The Stinger roadmap

Olson acknowledged that the MapR-led Apache Drill project is cut from the same cloth as Impala (that is, being a Google Dremel clone designed specifically for Hadoop), but “the difference is we’re shipping code.” Being generally available and ready for production workloads means Cloudera can lock down users and market share before many even have a chance to experiment with Drill. He all but dismissed questions over the readiness of Impala, spurred by rumblings in the Hadoop space that Cloudera rushed it into public beta in order to get on the scoreboard against more fully baked offerings.

“I don’t feel we’re under the gun competitively to pull it out of beta because no one else has product in the market,” Olson said. “I have no problems … calling this GA quality.” He did, however, acknowledge that Impala is shipping with a “minium viable feature set” that the company has plans to build on in the near future. Impala Senior Product Manager Justin Erickson noted a few issues of concern, including around the number of concurrent users Impala can support, but said they have been addressed during the beta period.

One piece of a larger platform

Really, though, the whole point of Impala and its competitors is to turn Hadoop from a tool for batch analytics and mass storage into a platform that can handle nearly all of companies’ data-processing needs. In that regard, it appears we’re just getting started. Cloudera, MapR, Pivotal Greenplum and Hortonworks are already pushing their own products and projects, and Olson said “it’s absolutely our intent” to enhance Cloudera’s platform with even more open-source products — perhaps even more database technologies a la HBase — that will let users do more stuff with more types of data. Over time, this strategy could result in Hadoop displacing the current breed of databases and data warehouses and becoming the single data store atop of which users run whatever applications they so desire. For now, though, especially when it comes to Impala and the data warehouse incumbents, Olson is taking a measured approach. “The likelihood that we’re going to knock them off in the near term,” he said, “… it would be a tough fight to win.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: Cloud and big data woo enterprises
• 2012: The Hadoop infrastructure market booms

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q2: Big data and PaaS gain more momentum

MapR’s decision to open source parts of it code is significant, but not groundbreaking. The company is only releasing its improvements to a handful of Hadoop-related Apache projects that are included in the MapR distribution of Hadoop, but not the proprietary code that’s MapR’s real competitive advantage in the contentious Hadoop market. While it’s still not flying the all-open-source banner like Hortonworks is, the code release puts MapR more on par with competitor Cloudera, which bolsters its open source aspects with some proprietary software for managing Hadoop clusters.

MapR also took another step in the open source direction on Thursday, announcing a partnership with Canonical that integrates MapR’s M3 distribution with the Ubuntu Linux operating system. The two also have plans to ease the installation of MapR’s Hadoop software on OpenStack-based cloud infrastructure.

I wrote recently in relation to MapR’s $30 million VC investment that the company is in a tricky position when it comes to open source. The Hadoop ecosystem was built on open source and still values it immensely, but some customers are definitely willing to pay money for products that deliver the features they want, open source or not.

As for Yelp, well, it’s just following in the footsteps of many companies — Netflix and everyone doing something on Kaggle (including GigaOM) — in trying to find new ways to use its data. The data set it’s releasing is from the Phoenix, Ariz., area and include 11,537 businesses, 8,282 checkin sets, 43,873 users and 229,907 reviews. The deadline for entries is May 20, and they can be submitted in pretty much any form you can imagine.

Hopefully, for Yelp’s sake, it doesn’t step in it the way other companies — including Netflix and AOL — have when they released supposedly anonymous data sets that were later de-anonymized. Releasing data sets gives clear benefits to both the source companies and institutions or individuals accessing the data, but privacy snafus have a away sneaking up and mitigating some of the goodwill.

Feature image courtesy of Shutterstock user Jakub Krechowicz.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• How search can unlock the power of big data
• Infrastructure Q1: Cloud and big data woo enterprises

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 2012: The Hadoop infrastructure market booms
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Because its roots are as an open-source project, some members of the Hadoop community are rightfully concerned about keeping it as open as possible. This gives customers more flexibility in moving from product to product, they argue, and could help prevent a technological splinter like what happened with Unix in the 1980s and significantly slowed the popular operating system’s uptake and rise to ubiquity.

MapR’s feature list

MapR catches some flak because it has made its name pushing a pair of Hadoop distributions (one free and one not) that are based on the company’s proprietary file system that it claims is significantly faster than the standard Hadoop Distributed File System that many of its competitors use. Last year, it announced a commercial version of the usually HDFS-based HBase database, currently in beta, that also includes many of MapR’s homegrown improvements around performance and reliability.

Although, according to MapR VP of Marketing Jack Norris, the criticisms of its semi-proprietary aren’t entirely fair. He told me during a recent call that there are more than a dozen open-source packages within the company’s Hadoop distribution, and noted that allowing data access via NFS is hardly a tool of vendor lock-in.

The company is also spearheading the Apache Drill project, an open-source re-envisioning of Google’s Dremel for SQL-like queries on Hadoop data. Tomer Shiran, MapR’s director of product management, will be discussing the project during a panel at our Structure: Data conference this week in New York.

But at the end of the day, MapR is a business and it’s doing what it can to make money in the new world of big data. If customers want features they can’t get from open-source versions of Hadoop, MapR will gladly supply them. In fact, he said, open source is “really not a core issue that comes up during the sales cycle.” (Norris took a more-defensive tone in a discussion about this topic last year: “No one can name the top 5 or 10 engineers on Oracle’s database,” he told me, “and no one really cares.”)

Norris points to a recent blog post from Gartner analyst Merv Adrian in defending his company’s position. Addressing the concern over open source and Hadoop — particularly as it relates to MapR and former OEM partner EMC — Adrian wrote: “Having some components of your solution stack provided by the open source community is a fact of life and a benefit for all. So are roads, but nobody accuses Fedex or your pizza delivery guy of being evil for using them without contributing some asphalt.”

But MapR could just as easily point to its customer list and partnerships to prove the effectiveness of its approach, at least. Norris said its customers in fields such as advertising and retail analyze data on more than 90 percent of the internet population monthly and more than a trillion dollars in transactions every year. (It’s pretty mum on naming customers, although Norris did cite ComScore and Ancestry.com as users.) Both Amazon Web Services and Google have partnered with MapR to boost Hadoop performance on their cloud platforms.

Still, Hadoop is still relatively young as a commercial technology and it’s very early on for Hadoop as an IT market all its own. What customers like now might not be what they like forever, and there’s plenty of competition for those workloads and dollars. When you look at its bigger, better-funded and better-known competitors such as Cloudera, Hortonworks, EMC Greenplum and now Intel, it’s easy to see just how tough a fight MapR has in front of it.

Norris isn’t sweating it, though. “The big major weakness that needs to be addressed [with Hadoop] is the dynamic read/write capability of HDFS,” he told me. As long as the other players keep relying on HDFS at the storage layer, MapR will at least have a strong point of differentiation.

Mayfield Fund led MapR’s latest investment round, and existing investors Lightspeed Venture Partners, NEA and Redpoint Ventures also participated.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How search can unlock the power of big data
• Scaling Hadoop clusters: the role of cluster management
• Infrastructure Q1: Cloud and big data woo enterprises

• EMC Greenplum lays down the SQL-on-Hadoop gauntlet: The company’s new Pivotal HD Hadoop distribution fuses its analytic database technology with Hadoop to create a single data store for everything. Greenplum co-founder Scott Yara claims the data warehouse — where Greenplum got its start — is the new mainframe.
• Intel does Hadoop: Intel’s Hadoop distribution is interesting for so many reasons, but the biggest might be the sense that it’s an attempt to keep x86 relevant as ARM pushers pursue big data workloads. Among Intel’s hardware partners are Cray, SuperMicro and Cisco.

• How Friendster died and Facebook might die: Researchers studied the collapse of Friendster and decided that a dimished cost-benefit analysis and users’ average number of friends contributed to its demise. The fewer friends, the more influential one friend’s decision to quit. And people quit when services begin to suck.
• Using memristors to recreate the brain: This is a heady research project based on the theory that memristors are similar enough to synapses in the human brain that they could help create an artificial brain. Memristors are a nanotechnology that allow electrical currents to pass between circuits based on the past currents they have  transmitted.
• MapR and Google in a high-performance lovefest: MapR is all about faster Hadoop, and Google is all about touting how great its Compute Engine cloud is for high-performance job. A MinuteSort benchmark test of MapR on Compute Engine bested the previous record (and crushed the previous Hadoop record for MinuteSort) — and on standard cloud servers, no less.
• LinkedIn open sources Databus: Databus is LinkedIn’s tool for updating changes in data between its various storage systems and applications at high speed. It could be pretty valuable, and I assume it’s something LinkedIn’s Bhaskar Ghosh will discuss during our guru panel at Structure: Data next month.

• Continuuity free beta now open to the public: Continuuity is the startup from former Yahoo VP Todd Papaioannou and Facebook engineer Jonathan Gray that’s building a platform as a service for developing big data applications. On Wednesday, it opened a beta version to developers who want to test the experience of building Hadoop applications on the cloud-based platform.
• Placed Analytics shows who shops in stores but buys online: This is the latest piece of research from Placed, a startup tracking mobile phone data to determine what businesses people like to visit, or at least hang out near. This report highlights which businesses are most at risk from consumers viewing products in their stores and then buying them on Amazon.
• IBM, South Korea and weather predictions:Weather forecasting has always been a good area for big data and high-performance computing, so this use case is pretty much straight data porn. From the press release: “IBM has provided KMA and NMSC with the latest IBM storage technologies capable of recording 20 gigabytes (equivalent to 400,000 web pages) of data per second … [w]ith a total storage capacity of 9.3 petabytes.”
• Virtustream using Druid for cloud analytics service: Virtustream is dead serious about staking its claim as theenterprise cloud provider, and this partnership with Metamarkets (see disclosure) is a good way to expand its reach into big data applications. Essentially, Metamarkets will provide consulting services for companies wanting to build apps atop Hadoop and Druid, the in-memory analytic database that Metamarkets created.

In addition to LinkedIn’s Ghosh, the founders of Placed, Continuuity and Metamarkets will all be on stage at Structure: Data talking about everything from building Hadoop applications, to managing massive data infrastructure to the new era of web privacy, so please come come and watch.

Disclosure: Metamarkets is a portfolio company of True Ventures, which is also an investor in GigaOM. Om Malik is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• How search can unlock the power of big data
• Infrastructure Q1: Cloud and big data woo enterprises

In a conference held in San Francisco, VP and General Manager of Intel’s Datacenter Software Division Boyd Davis explained Intel’s history in Hadoop that stretches back to 2009 and stressed that Intel is going to share some aspects of its Hadoop distribution, but not all. Intel has a distribution of Hadoop it has released in China, but today it’s bringing it to the United States Intel’s version of the Hadoop distribution uses Hadoop 2.0 and YARN, which is a cutting-edge version of  platform compared with what most Hadoop users have deployed thus far.

Why Intel wants to push its own version of Hadoop

Boyd introduced partners such as and Cisco, which has tuned the Intel Hadoop distribution for its own servers. Intel also hosted a panel that included executives from SAP, Red Hat and Savvis to discuss the challenges of big data and the promise of Hadoop and big data.

Davis was up front about Intel’s rationale for releasing its own distribution, namely that it was worried about the fragmentation and possible uncertainty associated with current Hadoop distributions. That could be read as a dig against the many startups already offering Hadoop distributions, all of which are slightly different (of course, Intel’s will be slightly different, too). Like all of the existing players such as Cloudera and MapR, Intel will open source certain aspects of its distribution, but will also keep software to itself.

Inside the data center, it’s no longer just web servers that matter

For example, Davis stressed that Intel will not share its management and monitoring software, which could be highly valuable for enterprise customers. The Intel software could coordinate with Intel’s data center management software and make managing a variety of workloads easier. And hidden in that coordination might be one Intel’s aims in pushing its own version of Hadoop — the threat of ARM chips used in Hadoop clusters.

Dell, Calxeda and others are evaluating the use of lower-performance, lower-power chips in Hadoop clusters, a market Intel would hate to cede in the data center as data grows and analytics becomes more important. To that end, Intel has also optimized its Hadoop distribution for solid-state drives, something that other Hadoop companies haven’t done so far.

When asked about Atom and the use of lower-performance processors for Hadoop, Davis noted that while people are using lower-end processors for Hadoop , but that those uses tend to have slower networking. Davis says that when you combine high-end processors with 10 gigabit Ethernet and Hadoop, customers get the performance that they want.

So while Intel may tout stability and consistency as the reason for it’s decision to become a major player in the software market for big data, it’s also driven by the changes in the data center that threaten the grip Intel has on the hardware inside the data center. The cloud and big data has changed the workloads and hardware requirements for the data center and Intel is playing the long game in trying to release software that can be tuned to its chips.

The Hadoop drama isn’t over yet

Intel isn’t the only big vendor touting its own homegrown version of Hadoop. On Monday, EMC’s Greenplum division announced an entirely revamped version of its Hadoop distribution that’s merged with it’s flagship analytic SQL database. These big companies have big existing businesses to protect and lots of resources to put into doing it. As my colleague Derrick Harris wrote on the EMC news:

Looking past his competitive boasting, though, it’s easy to see [Greenplum's Scott] Yara’s greater point when you ask him what all this Hadoop talks means for the data warehouse business on which Greenplum was built. He points to the mainframe business that fell from its high perch decades ago but still drives billions a year in revenue. A single MPP database system is still faster on certain workloads than SQL on Hadoop, but that gap will close over time and “I do think the center of gravity will move toward HDFS,” he said.

Hadoop is a juggernaut when it comes to big data. Intel is a juggernaut when it comes to data center infrastructure. Its decision to enter into the open source software market is a big one for the chip company, for the Hadoop ecosystem and for the myriad startups playing in this space. It’s a topic we’ll explore more during our Structure Data conference in New York on March 20 and 21.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Why service providers matter for the future of big data

• Cyber 9-11 threat could be imminent says Homeland Security Secretary Janet Napolitano.
• the Pentagon will quintuple its cybersecurity forces in response to fears of a “Cyber Pearl Harbor”; 
• Your own Linksys or D-link router may have betrayed you to hackers.

It’s enough to make you unplug, if that were an option.

But big security vendors are scrambling to meet the threat (and take advantage of these headlines) by touting how they are incorporating big data smarts into their latest offerings. RSA on Wednesday announced RSA Security Analytics -- which brings together the company’s NetWitness forensics and Envision log management and the GreenPlum MapR distribution, according to Amit Yoran, SVP of and GM of RSA’s Security Management & Compliance Business.

Amit Yoran, SVP and GM of RSA’s Security Management & Compliance Business Unit .

IBM’s on board with IBM Security Intelligence with Big Data, that ties together IBM Security QRadar SIEM and Big Data Platform among other things. Sandy Bird, CTO of IBM’s security systems division told the Wall Street Journal that this software can help “CIOs detect internal and external security threats in new ways—and can even scan email and social media to flag apparently ‘disgruntled’ employees who might be inclined to reveal company secrets.”

And Cisco Systems is buying Cognitive Security, a privately-held network security company that uses artificial intelligence to detect advanced cyber threats, according to the Cisco blog announcing the acquisition.

Big opportunity, crowded market

Big data security analytics, says Jon Oltsik, senior principal analyst with Enterprise Strategy Group, is obviously a hot market. Other contenders include SAIC and startups like RedLambda, PacketLoop, ZettaSet, Sumo Logic and  Palantir Palentir. The PacketPig open source project focuses on security and analytics, he said.

All these players see a chance to apply big data expertise to tilt the balance of power against cyber villains and because of big data’s ability to handle streams of log data and credentials, it does change the game, making it easier to prioritize threats and examine user behavior to spot anomalies, as GigaOM’s Derrick Harris has reported.

Traditionally, companies found threats by examining server and network logs to tell them if their systems had been accessed. It was a rear-guard, after-the-fact reaction. Proper use of big data analytics can accelerate the process. The idea is to apply analytics to this data as  it flows into the network and watch for patterns. “That lets you spot things that you would not have seen till after the fact before or would have thought was a blip,” said Judith Hurwitz, president of Hurwitz & Associates, an IT consultancy. The upshot is companies can shut threats down faster or possibly even prevent a breach by detecting hacker activity before it gets expensive.

Goal: better, faster info on digital evil doers

These new offerings promise to give companies a full picture of who’s coming into their network, who’s talking to whom, and spot anomalies or atypical user behavior while it is still actionable.

“If Johnny used five IP addresses and four user IDs and 12 different accounts  … the time to detect that activity will go from many hours to a few minutes.  This is a simple use case, not sci-fi.”  said Paul Stamp, director of product marketing for RSA.

Of course, this is an arms race. The good guys build and deploy cool new technologies, then the bad guys — no fools — use the same types of technology to overcome those defenses.

RSA’s executive chairman Art Coviello conceded as much at a press briefing Tuesday at RSA headquarters in Bedford, Mass. “It’s not about perfect security, its all about ratcheting down risk as much as you can.”

To learn more about the opportunities and challenges in the era of big data, check out GigaOM’s Structure Data event March 20-21 in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• How search can unlock the power of big data
• Scaling Hadoop clusters: the role of cluster management