Exclusive: Pivotal CEO says open source Hadoop tech is coming

10 Comments

Credit: Pivotal CEO Paul Maritz / Jakub Mosur

Pivotal, the cloud computing spinoff from EMC and VMware that launched in 2013, is preparing to blow up its big data business by open sourcing a whole lot of it.

Rumors of changes began circulating in November, after CRN reported that Pivotal was in the process of laying off about 60 people, many of which worked on the big data products. The flames were stoked again on Friday by a report in VentureBeat claiming the company might cease development of its Hadoop distribution and/or open source various pieces of its database technology such as Greenplum and HAWQ.

Gigaom has confirmed at least part of this is true, via an emailed statement credited to Pivotal CEO Paul Maritz:

“We are anticipating an interesting set of announcements on Feb 17th. However rumors to the effect that Pivotal is getting out of the Hadoop space are categorically false. The announcements, which involve multiple parties, will greatly add to the momentum in the Hadoop and Open Source space, and will have several dimensions that we believe customers will find very compelling.”

Those announcements will take place via webcast.

Paul Maritz at Structure Data 2014. (© Photo by Jakub Mosur).

Paul Maritz at Structure Data 2014.

Multiple external sources have told Gigaom that Pivotal does indeed plan to open source its Hadoop technology, and that it will work with former rival (but, more recently, partner) Hortonworks to maintain and develop it. IBM was also mentioned as a partner.

Members of the Hadoop team were let go around November when active development stopped, the sources said, and some senior big data personnel — including Senior Vice President of R&D Hugh Williams and Chief Scientist Milind Bhandarkar — departed the company in December, according to their LinkedIn profiles. Both of them claim to be working on new startup projects.

When EMC first introduced its Hadoop distribution, called Pivotal HD in February 2013 (it was one of the technologies that Pivotal the company inherited), executive Scott Yara touted the size of EMC’s Hadoop engineering team and the quality of its technology over that of its smaller rivals Cloudera, MapR and Hortonworks. However, Pivotal has been getting noticeably more in touch with its open source side recently, including with the Hortonworks partnership referenced above (around the Apache Ambari project) and a big commitment to the open source Tachyon in-memory file system project.

The current Pivotal HD Enterprise architecture.

The current Pivotal HD Enterprise architecture.

Pivotal has been a big proponent of the “data lake” strategy whereby companies store all their data in a big Hadoop cluster and use various higher-level programs to access and analyze. Last April, the company took a somewhat brave step toward ensuring its customers could do that by relaxing its product licensing and making Pivotal HD storage free.

Whatever happens with Pivotal’s technology, it’s not shocking that the company would decide to take the open source path. Its flagship technology is the open source Cloud Foundry cloud computing platform, and Cloudera, Hortonworks and MapR have cornered the market on Hadoop sales, by all accounts. If Pivotal has some good code in its base, it’s probably best to get it into the open source world and ride the momentum rather try to fight against it.

For more on the fast-moving Hadoop space, be sure to attend our Structure Data conference March 18-19 in New York. We’ll have the CEOs of Cloudera, Hortonworks and MapR on stage to talk about the business, as well as Databricks CEO Ion Stoica discussing the Apache Spark project that is presently kicking parts of Hadoop into hyperdrive.

10 Comments

Eli Cummings

When a company says we are going to open source our technology it usually means their business model around that technology has failed. It’s lipstick on a pig.

Steve Ardire

Gee thx for corroborating what I already stated that Derrick Harris agreed more readily with ;-)

Steve Ardire

Idiot read the rest of my post

Pivotal cannot compete with Cloudera, Hortonworks, MapR and should have taken a hint last year when Intel last year dropped its own commercial Hadoop distribution and threw its support behind market leader Cloudera along with $740M investment for 18% of them.

IBM still has their own Hadoop distribution but hardly gets any attention because they’re promoting the heck out of @ibmwatson ;-)

Steve Ardire

Gee Derrick one a rather lame post which is atypical of you!

Pivotal CEO Paul Maritz is blowing smoke up peoples arses because open source Hadoop is already here Pivotal is #latetotheparty

Pivotal cannot compete with Cloudera, Hortonworks, MapR and should have taken a hint last year when Intel last year dropped its own commercial Hadoop distribution and threw its support behind market leader Cloudera along with $740M investment for 18% of them.

IBM still has their own Hadoop distribution but hardly gets any attention because they’re promoting the heck out of @ibmwatson ;-)

Derrick Harris

Thanks for the backhanded compliment ;-) I believe I said as much in the post re: competing w/ Cloudera et al.

Steve Ardire

Yes sort of but you had to play through Pivotal CEO Paul Maritz blowing smoke up peoples arses because open source Hadoop is already here and Pivotal is #latetotheparty because they have been a sponsor of Giga conferences ;-)

Derrick Harris

I’m slightly offended by that accusation, but not severely offended because it’s meritless. That Hadoop is already open source or that Cloudera et al rule that market isn’t news; the news is that Pivotal is changing its strategy, for better or worse.

There are plenty of opportunities to read between the lines as to why it made that decision.

Steve Ardire

Pivotal’s open source side strategy will be a failure because there’s nothing compelling and you can get all the green components they have ( per your diagram ) elseware.

Paul Maritz is slick but smug and #latetotheparty

Ask GE ( put $105M Into Pivotal ) what they think about Pivotal changing its strategy.

Comments are closed.