Comments on: As Big Data Takes Off, the Hadoop Wars Begin

By: Joan E. Beaudoin

Joan E. Beaudoin — Wed, 30 Nov 2011 15:23:32 +0000

As Big Data Takes Off, the Hadoop Wars Begin – http://t.co/Ut2kKpqk

By: Charles Papir

Charles Papir — Tue, 29 Mar 2011 19:51:11 +0000

One of the must effective and longest longevity “Big Data” company I ever came accross has been 1010data LLC (1010data.com) and no one has ever mention them in any of their articles. The depth of experience and markets is impressive.

By: Grant Ingersoll

Grant Ingersoll — Tue, 29 Mar 2011 13:13:48 +0000

Just to be clear, Yahoo! never opensourced Hadoop. Hadoop was started by Doug Cutting and Mike Cafarella as part of the Apache Nutch project (an Apache Lucene related project) to help solve scaling out large web-crawling jobs and it was done well before Doug ever joined Yahoo!

By: Derrick Harris

Derrick Harris — Mon, 28 Mar 2011 18:14:39 +0000

Joachim,

I think you cut off the Red Hat analogy too soon. If Cloudera is Red Hat for Hadoop, one might call IBM the Novell of Hadoop, and DataStax the Ubuntu of Hadoop. Point being, demand is creating a number of suppliers all vying to be the OS, if you will, of Hadoop environments.

By: Anthony Kline

Anthony Kline — Mon, 28 Mar 2011 18:12:45 +0000

Derrick — great stuff on this article. It’s awesome to see Cloudera get this sort of attention. Big Data should be the focus of technology shops, vc’s, and entrepreneurs as we move into the future; it’s just a bigger problem than people recognize. Kudos to Amr & Jeff @ Cloudera for building a great team and having the vision to see how “big” Big Data will be in the years to come.

By: Derrick Harris

Derrick Harris — Mon, 28 Mar 2011 18:09:47 +0000

Duly noted. I call them alternatives because they give users an alternative option to writing directly in MapReduce. To the degree they simplify the experience for certain users, they might minimize the importance of any given MapReduce implementation or proprietary query language.

By: Owen O'Malley

Owen O'Malley — Mon, 28 Mar 2011 16:09:55 +0000

Pig and Hive are both Apache projects. They (and Cascading) build on top of Hadoop MapReduce to provide higher level abstractions rather than being alternatives. 75% of MapReduce jobs at Yahoo are in fact Pig jobs and 90% of MapReduce jobs at Facebook are Hive jobs.

By: Joachim Hubele

Joachim Hubele — Mon, 28 Mar 2011 15:24:26 +0000

Good overview, however I strongly disagree with the term Hadoop War.

- Hadoop is not a finished product, it never made to Release 1. It got basically stalled at 0.20.2 or 0.21.0, depending who you ask.
The reason is, that Hadoop is an extremely complex product. There are so many open bugs, that even if they are solved, it would be a huge effort to test the whole system again and verify still everything works.
No one will do this right now.
- There is high demand for a solution like Hadoop, just because of the growth of the internet. And, Hadoop being a complex product, users need support and guidance to use it.
It just a question of supply and demand that companies like Cloudera and DataStax are coming up. I would not say this a war. We have Red Hat Linux because there was a demand for a stable Linux with support. Nobody calls this a war …

By: SteveL

SteveL — Sun, 27 Mar 2011 17:20:52 +0000

-You can have other filesystems to HDFS, but how close do they come in $/TB for reliable storage. The strength of HDFS is not just its scale, but the cost of that scale: EMC will find that hard to compete against. Every terabyte stored in HDFS is a terabyte that EMC or IBM GPFS don’t get you paying for in hundreds of dollars.

By: Amr Awadallah

Amr Awadallah — Sat, 26 Mar 2011 04:20:07 +0000

Derrick, love the Flickr photo you picked :)