GigaOM » Datameer

Want to ditch your data scientists? Here are 7 startups that can help

Stacey Higginbotham — Thu, 05 Jul 2012 21:59:04 +0000

Ford’s data chief joined many other top executives who are bemoaning the lack of simple tools to solve big data problems — namely the fact that running Hadoop clusters or performing analytics is still a job that requires a specialist. If we want to big data revolution to scale, then we need to make it as easy as Netscape made the web surfing experience. Here are 7 startups making that happen.

Ford’s data chief John Ginder, did an interview with ZDNet in which he says:

“That’s a great endpoint I’d love us to move toward,” said Ginder, “but there aren’t enough of us and there aren’t enough of those tools out there to enable us to do that yet. We have our own specialists who are working with the tools and developing some of our own in some cases and applying them to specific problems. But, there is this future state where we’d like to be where all that data would be exposed. [And] where data specialists — but not computer scientists — could go in and interrogate it and look for correlations that might not have been able to look at before. That’s a beautiful future state, but we’re not there yet.”

Datahero: This startup is all about visualization — namely making it easy to take data and turn it into pretty pictures that can then generate new understanding or convince someone to take action. Users bring their datasets files and Datahero does the rest.

Prior Knowledge: Relative newcomer Prior Knowledge is the brainchild of MIT grads who wanted to let non data scientists play around with data. The company offers a service that lets people upload their data and hook into PK’s database API. The service then assess the information for correlations as well as helps app developers build predictive models. It’s raised $1.4 million in funding from Founders Fund and angels.

Platfora: Hadoop is everyone’s favorite big data batch processing platform, but it’s not easy enough for everyone to use. Like others Platfora wants to make Hadoop so easy even I could use it, through an intuitive user interface that has advanced data science functions built in, rather than making users perform queries. It has raised $5.7 million and its product will be out next year.

ClearStory: Big names back this startup, which is also a service as opposed to software. Google Ventures, Andreeseen Horowitz, and Khosla Ventures have funded ClearStory, which aims to help funnel data from a variety of source (including Hadoop!) into one place, where employees can then use a GUI to interact with and visualize that data.

Karamasphere: The Karmasphere product is designed to ease the process of developing Hadoop workloads and applications, even from the desktop. It lets users write SQL-like queries while also connecting to their favorite BI tools and analytics software to the software to perform analysis.

Datameer: Like others on this list Datameer is out to make Hadoop more relatable to non nerds. In this case it does this by creating a familiar spreadsheet overlay so businesspeople can analyze their Hadoop jobs and then let’s people create visualizations and draw correlations. It’s closest to Karamsphere, but its latest feature that allows someone to run it on a single machine is a differentiator.

BigML: Much like Prior Knowledge, BigML is a startup that combines data with machine learning to help give normal people access to the smarts to help them answer questions with their data. It hopes to let people do machine learning in four easy steps: set up a data source; create a dataset; create a model; and generate predictions. It’s in private-beta mode right now.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Who’s connected to whom in Hadoop world [infographic]

Derrick Harris — Fri, 29 Jun 2012 19:11:33 +0000

To say there are a lot of companies involved in the Hadoop ecosystem would be an understatement. To say partnership strategies are broad would be one, too. The folks at Datameer created this infographic to show just how expansive and interconnected the Hadoop ecosystem is (and, admittedly, to show off the new infographic tools in the latest Datameer release). When I say that Hadoop is poised to explode because everyone company under the sun has committed to it, here’s what I mean (and this data is even a bit outdated; there would be even more connections today).

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Is 2013 the year Hadoop uptake turns into a ‘tornado’?

Derrick Harris — Mon, 11 Jun 2012 18:38:12 +0000

Karmasphere CEO Gail Ennis told me recently she thinks “2013 is going to be the year when we see [Hadoop adoption] go a lot more mainstream and [turn] into a tornado.” I like the prediction, as much for its imagery as for her near-term certainty. Hadoop right now is like a funnel cloud spiraling above the ground: it took some strong forces (famous users, lots of hype and lots of venture capital) to create the cloud, but it’s still too high up to do real damage. When it hits the ground, though, watch out.

Ennis and her company Karmasphere are among the companies trying to bring Hadoop down to ground level or, in corporate IT parlance, to business users rather than data scientists and systems engineers. The easier Hadoop is to use, the more insights companies get to discover and the easier it is to justify paying for it.

Breaking down (or setting up) barriers

Karmasphere has been trying to help data analysts build better Hadoop applications for a couple years, but on Monday it took a big step forward with the 2.o release of its namesake software. The new version focuses on self-service and collaboration, letting business analysts, data analysts and others share a workspace in which they can team up on datasets to make the most of their individual skills. Most importantly, though, they can do so without calling in the IT department for help accessing their data or running a job.

It’s a trend we’ve reported on numerous times — one that Karmasphere and Datameer helped kick off, and that has since been given new life (GigaOM Pro subscription req’d) by Microsoft and a slew of other startups ranging from Platfora to Qubole. That’s a good thing for numerous reasons, including that business users and IT personnel are getting sick of each other, Ennis said — “IT wants to teach users to fish.”

When it’s easier to access and analyze data stored in Hadoop (Karmasphere 2.o, for example, lets users write SQL-like queries while also connecting to their favorite BI tools and analytics software), business departments and IT both get a little more space. That’s also means it’s easier to sell Hadoop products because everyone knows their roles and what items are coming out of whose budgets. Individual departments buy and manage business tools such as Karmasphere, while IT buys and manages system software such as Cloudera or Hortonworks.

You scream, I scream, we all scream for Hadoop

Datameer, is also doing its part to make the Hadoop tornado touch down, announcing on Monday a new version of its spreadsheet-for-Hadoop product that runs on a single computer. Among the usability and data-access improvements to its own 2.0 release (such as a WYSIWYG infographic creator) are a Workgroup edition that resides on a single server and supports up to 1TB and 50 users, and a personal edition that installs on a laptop or desktop computer.

Such small deployments might seem like an oxymoron — a single server, much less a single laptop is hardly big data — but big data is a misnomer itself. The real benefit of technologies such as Hadoop is being able to analyze new types of data in new ways, regardless how much you have. Small companies, departments or even individuals might not need a whole Hadoop cluster, but still might want to analyze their unstructured data in ways that other BI tools won’t let them.

Datameer’s personal edition is particularly compelling because of the flexibility it allows. In the case of one beta customer, VP of Marketing Joe Nicholson told me, an employee was able to import some data onto his desktop and, in a few nights working at home, draw a correlation between temperature and the amount of ice cream and water his company sells. The company had been struggling with this correlation for a while, but because of Hadoop the employee was able to analyze the company’s traditional sales data against publicly available weather data.

Given that it’s halfway through 2012, Ennis’s prediction of 2013 being the year that Hadoop turns into a tornado and sucks up everything its path seems fair enough. What Karmapshere and Datameer are doing — along with the availability of antipicated products from Platfora, Continuuity and other stealth-mode startups, cloud-based Hadoop services, and non-stop innovation on at the distribution layer — will make Hadoop a lot harder to resist, at least.

Feature image courtesy of Shutterstock user Melanie Metz.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The importance of putting the U and I in visualization

Derrick Harris — Fri, 04 May 2012 06:55:34 +0000

Ask a venture capitalist about big data and she will probably tell you about visualization. Only it won’t be visualization in the usual sense. Instead, it will be about visualization of the user interface. We’re talking about strikingly intuitive UIs that let users visually work with data using charts and tools instead of with algorithms and code. It’s hard work to do right — especially when you’re talking about massive data sets and complex computations — but the payoff could be huge for businesses.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

2012: The Hadoop infrastructure market booms

Thu, 26 Apr 2012 19:22:32 +0000

For years, technologists have been promising software that will make it easier and cheaper to analyze vast amounts of data in order to revolutionize business. More than one solution exists, but today Hadoop is fast becoming the most talked about name in enterprises. There are now more than half a dozen commercial Hadoop distributions in the market, and almost every enterprise with big data challenges is tinkering with the Apache Foundation–licensed software. This report examines the key disruptive trends shaping the Hadoop platform market, from integration with legacy systems to ensuring data security, and where companies like Cloudera, IBM, Hortonworks and others will position themselves to gain share and increase revenue.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

What it really means when someone says ‘Hadoop’

Derrick Harris — Mon, 06 Feb 2012 20:12:12 +0000

Big data is among the hottest trends in IT right now, and Hadoop stands front and center in the discussion of how to implement a big data strategy. There’s just one problem that keeps cropping up: many people don’t seem to know exactly what it means when somebody says “Hadoop.”

The problem surfaced again Monday in the form of complaints over Forrester’s new report titled “Enterprise Hadoop Solution, Q1 2012.” InformationWeek spoke with a few vendors that didn’t like how their products were assessed, and database industry analyst Curt Monash says the report “compares apples, peaches, almonds, and peanuts.” I thought the same thing when I saw a copy of the report last week. They all focus on Hadoop, but Hortonworks is not Datameer is not HStreaming.

Allow me to explain. Hopefully, this provides a foundation for parsing what people talk about when they talk about Hadoop, and for differentiating one type of product from another. (And you can learn even more about Hadoop and how it’s used at our Structure: Data conference taking place next month in New York City.)

What Hadoop is

I went into this in more detail in a GigaOM Pro report published last March (sub req’d), but the long and short is that Hadoop is, at its core, an Apache Software Foundation project consisting of two primary subprojects — Hadoop MapReduce and the Hadoop Distributed File System. MapReduce is the parallel-processing engine that allows Hadoop to churn through large data sets in relatively short order. HDFS is the distributed file system that lets Hadoop scale across commodity servers and, importantly, store data on the compute nodes in order to boost performance (and potentially save money). These are the two must-have components for any Hadoop distribution.

There are also a number of Apache projects related to Hadoop, often built atop either Hadoop MapReduce or HDFS. These include — but are not limited to — Hive and Pig, two SQL-like query languages to provide data-warehouse-like capabilities to a Hadoop cluster, and HBase, a NoSQL database that leverages HDFS as its distributed storage engine.

Hadoop distributions

These are packaged software products that aim to ease deployment and management of Hadoop clusters compared with simply downloading the various Apache code bases and trying to cobble together a system. Presently, Cloudera, Hortonworks, MapR and EMC all offer their own Hadoop distributions. Although they’re all unique — sometimes very unique, as with MapR’s proprietary file system — they all package a set of Hadoop projects (MapReduce, Hive, Sqoop, Pig, etc.) in a way that in theory makes them integrate more naturally, and to run both smoothly and securely.

Many Hadoop distributions integrate with various data warehouses, databases and other data-management products, with the goal of moving data between Hadoop clusters and other environments so each might process or query data stored in the other.

Hadoop management software

Just as the wording implies, Hadoop management software is designed to make it easier to manage and troubleshoot a Hadoop cluster. Such products are usually sold or offered by companies peddling Hadoop distributions, because even when commercially packaged, Hadoop is still a complex architecture and somewhat foreign to most IT personnel and products. However, third parties such as Platform Computing (now part of IBM) and Zettaset also sell software for managing Hadoop clusters, and their products are typically agnostic as to what distributions they support.

But distributions and management software are all about the infrastructure and the platform. Anyone actually wanting to use Hadoop still needs to know how to write applications that leverage the underlying architecture.

Hadoop application software (or, products that use Hadoop)

The Hadoop ecosystem gets really complex when we start looking at products that exist to help developers write Hadoop applications or otherwise analyze data stored within Hadoop in a manner other than writing traditional MapReduce jobs. These range from abstraction layers such as Karmasphere Analyst or IBM Infosphere BigInsights, to Hadapt, which offers a single-platform product fusing a SQL data warehouse with a Hadoop cluster, to HStreaming, which promises real-time processing and analytics.

The one common thing among all these products, however, is that they are not Hadoop distributions, but sit atop platform software from Hortonworks, EMC or whomever. Some products that get thrown into the Hadoop fray, such as Outerthought Lily or Drawn to Scale Spire, are essentially scale-out databases built atop HBase (which itself is a separate project built atop HDFS). The image below, from Karmasphere, gives a particularly clear map of how a Hadoop environment might look.

The applications and analytics space is probably where we’ll see the biggest influx of new companies, as writing Hadoop applications is still tough, but it’s also how companies will actually start experiencing direct business benefits. In fact, it’s these type of higher-level products that are the focal point of Accel Partners’ new big data fund.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

6 reasons why 2012 could be the year of Hadoop

Derrick Harris — Thu, 24 Nov 2011 14:02:36 +0000

Hadoop gets plenty of attention from investors and the IT press, but it’s very possible we haven’t seen anything yet. All the action of the last year has just set the stage for what should be a big year full of new companies, new users and new techniques for analyzing big data. That’s not to say there isn’t room for alternative platforms, but with even Microsoft abandoning its competitive effort and pinning its big data hopes on Hadoop, it’s difficult to see the project’s growth slowing down.

Here are six big things Hadoop has going for it as 2012 approaches.

1. Investors love it

Cloudera has raised $76 million since 2009. Newcomers MapR and Hortonworks have raised $29 million and $50 million (according to multiple sources), respectively. And that’s just at the distribution layer, which is the foundation of any Hadoop deployment. Up the stack, Datameer, Karmasphere and Hadapt have each raised around $10 million, and then are newer funded companies such as Zettaset, Odiago and Platfora. Accel Partners has started a $100 million big data fund to feed applications utilizing Hadoop and other core big data technologies. If anything, funding around Hadoop should increase in 2012, or at least cover a lot more startups.

2. Competition breeds success

Whatever reasons companies had to not use Hadoop should be fading fast, especially when it comes to operational concerns such as performance and cluster management. This is because MapR, Cloudera and Hortonworks are in a heated competition to win customers’ business. Whereas the former two utilize open-source Apache Hadoop code for their distributions, MapR is pushing them on the performance front with its semi-proprietary version of Hadoop. This means an increased pace of innovation within Apache, and a major focus on management tools and support to make Hadoop easier to deploy and monitor. These three companies have lots of money, and it’s all going toward honing their offerings, which makes customers the real winners.

3. What learning curve?

Aside from the improved management and support capabilities at the distribution layer, those aforementioned up-the-stack companies are already starting to make Hadoop easier to use. Already, Karmasphere and Concurrent are helping customers write Hadoop workflows and applications, while Datameer and IBM are among the companies trying to make Hadoop usable by business users rather than just data scientists. As more Hadoop startups begin emerging from stealth mode, or at least releasing products, we should see even more innovative approaches to making analytics child’s play, so to speak.

4. Users are talking

It might not sound like a big deal, but the shared experiences of early Hadoop adopters could go a long way toward spreading Hadoop’s utility across the corporate landscape. It’s often said that knowing how to manage Hadoop clusters and write Hadoop applications is one thing, but knowing what questions to ask is something else altogether. At conferences such as Hadoop World, and on blogs across the web, companies including Walt Disney, Orbitz, LinkedIn, Etsy and others are telling their stories about what they have been able to discover since they began analyzing their data with Hadoop. With all these use cases abound, future adopters should have an easier time knowing where to get started and what types of insights they might want to go after.

5. It’s becoming less noteworthy

This point is critical, actually, to the long-term success of any core technology: at some point, it has to become so ubiquitous that using it’s no longer noteworthy. Think about relational databases in legacy applications — everyone knows Oracle, MySQL or SQL Server are lurking beneath the covers, but no one really cares anymore. We’re hardly there yet with Hadoop, but we’re getting there. Now, when you come across applications that involve capturing and processing lots of unstructured data, there’s a good chance they’re using Hadoop to do it. I’ve come across a couple of companies, however, that don’t bring up Hadoop unless they’re prodded because they’re not interested in talking about how their applications work, just the end result of better security, targeted ads or whatever it is they’re doing.

6. It’s not just Hadoop

If Hadoop were just Hadoop — that is, Apache MapReduce and the Hadoop Distributed File System — it still would be popular. But the reality is that it’s a collection of Apache projects that include everything from the SQL-like Hive query language to the NoSQL HBase database to machine-learning library Mahout. HBase, in particular, has proven particularly popular on its own, including at Facebook. Cloudera, Hortonworks and MapR all incorporate the gamut of Hadoop projects within their distributions, and Cloudera recently formed the Bigtop project within Apache, which is a central location for integrating all Hadoop-related projects within the foundation. The more use cases Hadoop as a whole addresses, the better it looks.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Karmasphere pushes new big data workflow

Derrick Harris — Wed, 21 Sep 2011 17:15:07 +0000

Hadoop is all the rage in analytics, but it still isn’t easy for mere mortals to utilize the big data framework to its fullest extent. A handful of companies are trying to solve this problem by making it more intuitive to derive insights from Hadoop, including Karmasphere with the latest version of its Analyst Big Data product.

Unlike some Hadoop startups that target business users, Karmasphere is trying to woo data analysts who are well versed in working within data warehouses but still need some guidance to translate that knowledge into a Hadoop environment. To that end, the company has devised a workflow for accessing, assembling, analyzing and acting upon big data. At a high level, here’s how Karmasphere defines each of the steps:

Access – Connect to any Hadoop cluster on premises or in the cloud, adding metadata to describe Hadoop clusters and other data sources.

Assemble – Gather, organize and prepare any kind of data; unstructured, semi-structured, structured,
including compressed data formats, on-the-fly to perform analytics.

Analyze – Query and interact with data on Hadoop, learn and iterate to discover patterns and trends for
insight.

Act – Deliver actionable, transformative insights to people and businesses.

The company explains its new approach to Hadoop workflows in a white paper also published today.

For anyone wondering where Karmasphere, as well as similar startups such as Datameer and Platfora, fits into the Hadoop ecosystem, the graphic below is pretty informative. They’re on the top level for focused on analytics, while companies such as Cloudera, EMC Greenplum and MapR operate on the lower level with their Hadoop distributions that focus on cluster management and performance.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Infrastructure Q2: Big data and PaaS gain more momentum

Tue, 19 Jul 2011 07:01:55 +0000

Big data and Platform-as-a-Service offerings highlighted the second quarter, suggesting that we can expect to see a shift in enterprise IT practices around application development and analytics very soon. On the PaaS front, we saw new projects like DotCloud and Cloud Foundry gain incredible momentum in just a few short months. The big-data activity ranged from major new Hadoop vendors to heavy investment in flash storage that will speed the serving of data to processing engines. In other areas, we saw an uptick in cloud-computing plans from large vendors, OpenStack continued to mature and pick up both contributors and users, and Facebook caught our eye by launching an open-source project around the designs for its specialized servers and data centers. Additional companies mentioned in this report include VMware, Salesforce.com, IBM, Heroku and Calxeda. For a full list of companies, and to read the full report, sign up for a free trial.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Defining Hadoop: the Players, Technologies and Challenges of 2011

Wed, 30 Mar 2011 15:46:45 +0000

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.