6 Comments

Summary:

Commercial Hadoop startup Karmasphere today released the results of a survey of 102 Hadoop developers regarding adoption, use and future plans. The results provide some interesting insights into how Hadoop grows within organizations and underscore its status as an extremely valuable, but none-too-simple analytics tool.

Source: Karmasphere

From the I-was-going-to-conduct-this-research-but-someone-beat-me-to-it department, commercial Hadoop startup Karmasphere today released the results of a survey (PDF) of 102 Hadoop developers regarding adoption, use and future plans. The results provide some interesting insights into how Hadoop grows within organizations and underscore its status as an extremely valuable, but none-too-simple analytics tool. Of course, the latter characterization is why ISVs like Karmasphere, Cloudera and Datameer exist: to make millions by reducing the Hadoop learning curve.

Among the key results:

  • Sixty-eight percent of deployments begin as skunkworks projects, with 86 percent advancing to active development or production environments within a year.
  • The top three reasons for using Hadoop are data mining for business intelligence (19 percent), lowering the cost of data analysis (15 percent) and performing log analysis (13 percent), although uses like ETL (11 percent), scientific research (10 percent) and better utilizing unstructured data (9 percent) aren’t far behind. The longer organizations use Hadoop, the more valuable they find it and the more uses they find for it.
  • The number of Hadoop developers looks to rise by between 50 and 60 percent within the next year.
  • Java is the dominant language (86 percent), with Pig and Hive sharing the No. 2 spot at 44 percent each (multiple responses were allowed).
  • The steep learning curve (44 percent) and hiring qualified people (34 percent) top the list of general challenges, while debugging Hadoop jobs (63 percent) and monitoring Hadoop jobs (47 percent) top the list of programming challenges. Seventy percent of respondents feel that these challenges will have a major-to-moderate effect on growing or expediting their Hadoop deployments.

Based on what I’ve seen and heard about Hadoop, these numbers seem accurate. They’re also the reasons why the above-mentioned startups are receiving a lot of attention from all types of users and vendors, and why the ecosystem of commercial products supporting Hadoop keeps on growing. Hadoop is arguably the most mature tool for analyzing large volumes of unstructured data, and these numbers suggest it’s also the most capable. When commercial products evolve enough to mitigate the learning curve and overall lack of skills, watch out.

Related content from GigaOM Pro (sub req’d):

  1. “The longer organizations use Hadoop, the more valuable they find it and the more uses they find for it.”

    Ah, survivorship bias never dies! The less an organization finds Hadoop to be useful, the more quickly it will (and should) stop using it.

    Share
  2. normally the longer organisations are known to use hadoop. the more valuable it turns out to be, the more intense the use.

    Share
  3. [...] can also be said that lack of a vendor ecosystem definition is hurting adoption. A recent survey revealed that the largest challenge facing enterprises considering Hadoop is the steep learning [...]

    Share
  4. [...] Karmasphere isn’t yet a household name in the Big Data community, but it could be on its way. A recent survey showed a steep learning curve as the No. 1 impediment to Hadoop development, and this exactly the [...]

    Share
  5. [...] that requires a team of scientists to use and that only works for grand-scale problems. They want to see a product that the current IT staff can use today (or soon) for today’s problems, and they want to see proof that others have deployed it successfully [...]

    Share

Comments have been disabled for this post