Ex-Yahoo, Facebook big data vets do Hadoop for developers


Continuuity, a startup founded by former big data experts at Yahoo (s yhoo) and Facebook (s fb) is ready to share its Hadoop-based Platform as a Service with the world. Rather than simply providing another cloud service for writing and running Hadoop jobs, Continuuity wants to be the middle man between developers and the complex Hadoop clusters below their applications. It announced its existence, along with $2.5 million in funding from Battery Ventures, Andreessen-Horowitz and others, in January.

The company calls its product the AppFabric, and it’s essentially a software development kit and a set of high-level APIs sitting above a fabric layer that connects to all of a company’s various Hadoop clusters. Co-founder and CEO Todd Papaioannou, former entrepreneur in residence at Battery Ventures and VP and chief cloud architect at Yahoo, says Continuuity wants to make it easy for developers at fast-follower companies to build big data applications without having to learn Hadoop’s low-level APIs and the general complexity of the distributed framework. Very few people, he said, really want to be part of the “home-brew computing club experience.”

In some ways, Continuuity is trying to do for big data what Heroku (s crm) and other PaaS providers have done for web applications. There’s a drag-and-drop user interface, an integrated developer environment to make development easy on a laptop before pushing an application into production, and resources scale elastically.  Papaioannou said developers get a UI for DevOps, as well, and can connect Continuuity apps with server-side web apps wherever they’re hosted, even on other PaaS offerings.

Presently, Continuuity is available in private beta for its single-node developer version and private cloud version. It expects a public cloud version will be available at some point in 2013. The platform currently supports Java applications only — a natural first step given that Hadoop is written in Java — but Papaioannou said Continuuity plans to support other languages going forward.

Papaioannou’s fellow co-founder, Jonathan Gray, who was integral in building Facebook’s Messaging and Puma analytics applications on top of the Hadoop-based HBase database, says even early big data adopters might be interested in what Continuuity is selling. For example, he explained, a company like Facebook will have separate clusters for data warehousing, for HBase and for caching. A product like AppFabric would abstract the differences between those various systems and allow developers to easily write applications connecting to any or all of them.

Even if they’ve made it easy for non-engineer developers to write batch-processing jobs (like Facebook has with Hive for Hadoop), both Gray and Papaioannou say large companies can have hard time giving developers even more functionality. AppFabric might let developers create streaming or transactional apps, too, without requiring a whole new set of internally developed tools.

Really, Papaioannou says, the idea behind a PaaS for big data is no different than that behind platforms as a service, generally. The key question is: “Can a programmer sit down in an afternoon and develop something he can take to his boss?” Any company that can really make this happen for Hadoop — be it Continuuity, Infochimps, Mortar Data or whomever — should be in a good position to grow. Hadoop skills are in high demand but short supply, and there are plenty of developers waiting to get their hands on all that data-processing power.

Comments are closed.