VMware is charting its course using its virtualization prowess and Nicira’s software-defined networking, which it bought last year. It’s no surprise that VMware parent and storage leader EMC is pinning its strategy on software-defined storage technology that it’s calling ViPR.

The first software deliverables — and it’s all software built by a team led by Amitabh Srivastava, the former Microsoft cloud exec who joined EMC in 2011 – will be outlined Monday at EMC World in Las Vegas. But, here’s how Jeremy Burton, EMC’s CMO and executive vice president, outlined the plan for me on Friday.

First: A new control plane will let admins manage physical assets, including storage arrays, to create virtual arrays and storage pools and then provision them and make them available in a service catalog for users, Burton said. EMC likens the software to a universal remote control that can operate multiple devices. For most storage, ViPR will discover what storage assets are available and allow provisioning. And if there is “smart” storage available, it will offload processing to that array to handle the data path.

Second: A new data plane will initially focus on data objects — at first those stored in Amazon’s S3 by the third quarter and then  HDFS by year’s end, Burton said. ViPR data services will also support OpenStack Swift-compatible REST APIs as well as existing EMC Atmos and VNX storage and even storage from rival NetApp, Burton said.

From the press release: “For traditional workloads that utilize file and block [storage], EMC ViPR steps out of the way and lets the underlying array fulfill the role of Data Plane.” But for new web-scale, big data applications, it provides Object Data Services, which is where the support Amazon S3 and OpenStack Swift REST APIs and HDFS access methods comes in. Of course it will support existing NFS and iSCSI protocols that drive much of enterprise storage now. That heterogeneous  support means that customers can, depending on the workload type, configure their services as they see fit without necessarily  having to worry about what storage is under the covers.

IDC analyst Vernon Turner said EMC ViPR is about much more than “storage virt.” Instead, it “takes many of the elements needed to create the software-defined data center [including] wide-spread orchestration — so it can properly deliver cloud services.’

EMC, which made its fortune selling big, expensive storage hardware, has shown a willingness to cannibalize its own business before others can eat it for them. Burton said the design point was cloud, and the first targeted companies will be service providers, he said. “The design goal is no single point of failure … and ability to scale out,” he said. Service providers, including telcos and big hosting companies, are looking for ways to stave off incursions into their businesses by insurgents, especially Amazon Web Services.

Asked if EMC/VMware spinout Pivotal will use the new software as a foundational technology, Burton said the “Let me just say that Paul Maritz is one of Amitabh’s best friends right now.”

Maritz, who is Pivotal’s CEO, “wants a cheap massively scalable store for what he’s building,” Burton said. “I think Paul’s mission is to be cloud-agnostic but he also believes HDFS will be foundational layer and we intend to be primary provider of that layer.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: Cloud and big data woo enterprises
• AWS Storage Gateway jolts cloud-storage ecosystem

He pledged Wednesday that PivotalOne technology will run on Amazon Web Services, and on OpenStack and other infrastructures. And that this effort will be independent of its owners EMC and VMware (and now General Electric as well).

That is a very, very tall order, which I’ll get back to. But first, the actual news from the official Pivotal launch event:

• Maritz now has a title: he is Pivotal’s CEO. Up until now he was just sort of the “head” of Pivotal.
• The first deliverables of PivotalOne will be generally available in “less than 6 months,” according to Pivotal SVP Scott Yara. It was, however, unclear which parts he was referring to.
• We can now definitively drop the “Initiative” from the Pivotal Initiative name.

To be fair, Pivotal was just spun off four months ago, after six months of leaks. And in February it made its first move, launching Pivotal HD, a re-architected Hadoop implementation that is key to this whole effort.

How independent is independent?

Now, back to positioning.  Maritz, who spent 14 years at Microsoft, where he helped lead the Windows 95 and then Windows NT server charge, has huge credibility with developers and customers alike. If anyone knows from OSes — and what customers want from them — it’s him.

But the notion that Pivotal will be given free reign to cozy up to Amazon, or Microsoft, or other infrastructure providers as needed when it is still part-and-parcel of EMC and VMware — both of which have their own cloud agenda — strains belief. VMware is launching its own public cloud May 21.

Much of Pivotal’s promise — it will provide an abstraction layer atop third-party infrastructure so that applications will run anywhere — is what VMware promised with Cloud Foundry, the open-source PaaS that is now part of Pivotal. But there’s also the new Pivotal Data Fabric which builds on EMC’s GreenPlum technology and VMware’s Gemfire fast transaction processing capabilities. And then there will be Pivotal Expert Services, including Pivotal Labs’ rapid application deployment capabilities, which can be offered in conjunction with or atop the whole stack.

I don’t doubt the quality of the technologies here or even Pivotal’s ability to integrate them. But customers will wonder just how independent an entity that is 90 percent owned by two technology giants can really be to work with competitors in a scenario that will reassure corporate customers who don’t want lock-in.

The fact that GE is investing $105 million in this effort remains the headline. That news has got to have both Amazon and Microsoft sweating. Amazon is the world’s biggest public cloud provider and is adding more enterprise-class services by the day atop that cloud. Microsoft is pitching Azure now as an AWS infrastructure rival but has also tried to be the PaaS of choice for the millions of Windows businesses out there.

Both of those companies would have given a lot for an endorsement that GE gave Pivotal early today.

Check out Om Malik’s interview with Maritz earlier this year at our Structure:Data conference below:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• IT spending update, third quarter 2012
• Takeaways from the second quarter in cloud and data

Bill Ruh, GE’s VP, will talk about the investment and the Pivotal One Enterprise PaaS later today on Pivotal’s coming out party at 1 p.m. Eastern Time, joining Paul Maritz, the former VMware CEO (pictured above) who’s heading up Pivotal.

GE, the huge conglomerate that makes everything from household appliances to medical devices to jet engines, has been talking up the industrial internet of late. And now GE’s Global Software Center in San Ramon, Calif., will to use the Pivotal technologies as a standard way to deliver data management across GE units, according to the joint announcement.

According to statement released Wednesday morningt:

“The companies also announced their intent to enter into a broad research and development and commercial agreement aimed at accelerating GE’s ability to create new analytic services and solutions for its customers. The investment and business agreement are each expected to be finalized in the second quarter of 2013 and are subject to standard regulatory approval and other closing conditions.”

Update: With GE aboard, EMC will own 62 percent; VMware 28 percent and GE 10 percent, according to a Pivotal spokesperson. Previously  EMC owned 69 percent, VMware 31 percent. When the project  – which took on assets from both companies —  launched, it had 1,250 employees and represented about $300 million in revenue, EMC Chairman Joe Tucci said last month.

This report was updated at 9:22 a.m. PST with the new ownership stakes

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• A 2011 Infrastructure Forecast
• Takeaways from the second quarter in cloud and data

The Bellevue, Wash.-based early-stage VC firm will also open an office in Palo Alto, Calif. to better attack these opportunities, said Frank Artale, general partner who will run this new fund, informally dubbed Ignition V. The company brought on Nick Sturiale, a new partner, to run that office.

“We think that businesses and people who work in businesses have been largely underserved for the past 15 years,”  Artale said in a recent interview.

The goal of the dual offices is to promote cross-pollination and collaboration. ”We want to do real social networking here — not just Facebook stuff,” Artale added. “Palo Alto and the Bay Area are super important as great entrepreneurial engines — Cisco, Oracle and other companies down there spit out great entrepreneurs.”

Goal: Apps that combine consumer ease of use with enterprise utility

Ignition has some credibility in the enterprise. Several team members – including Artale, John Connors, and Cameron Myhrvold — are former Microsoft executives. And previous investments include Cloudera, Splunk, Zenprise, DocuSign, Opscode, Parse and Bromium.

New enterprise applications have to work well and look good on laptops and PCs, but also on tablets and phones as the consumerization of IT trend continues, he said.

Artale which described the new fund as “slightly oversubscribed”  took three months to fund. Investors include new and existing university endowments, pension funds and investment companies. Ignition V is smaller than the previous fund, which weighed in at $400 million but will also focus more — eschewing investments in telecom and consumer internet companies, Artale said.

The notion that enterprise IT is back as a hot category is cropping up all over. New vendors — large and small — are building consumer-grade products but for business use. Pivotal Initiative chief Paul Maritz spoke in depth about this at the recent Structure: Data conference in New York and the topic will doubtless crop up again at Structure in San Francisco in June.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• 2012: The Hadoop infrastructure market booms
• The fourth quarter of 2012 in cloud

Speaking with Om Malik on Wednesday in New York for GigaOM’s Structure:Data conference, Maritz talked about the two potential courses a company might take in heading toward a data-driven future:

“I think there’s two ways. One way is, unfortunately, Darwinian evolutions,” he said. “But others will change and change always comes through leadership, where someone in the organization has to really step up and drive change.”

Maritz said when he was at VMWare, he was fascinated that in looking at different companies in the same businesses, he could see that some would be fifty percent virtualized, and others only ten percent virtualized, simply because of the mindset at the organization that had inspired the change.

“Change really requires leadership. It requires people to understand the organization and get behind it,” he said.

Check out the rest of our Structure:Data 2013 coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Why the big data startup boom will likely be short-lived
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

The key is developing the tools and techniques to properly collect, harvest, and analyze that data. On Wednesday and Thursday this week, we’ve gathered some of the top minds in this emerging field in New York at Structure:Data to help you understand how data and data analysis can improve your business and your life. They include Paul Maritz, chief strategist at EMC and a leader of The Pivotal Initiative; data experts from companies like Facebook, Microsoft, IBM and a host of startups; and even the CTO for the Central Intelligence Agency, Ira “Gus” Hunt.

More information about the conference can be found here. If you can’t join us in New York, follow along with the livestream here, or join the conversation on Twitter with #dataconf. We’ll also post a roundup of all our coverage from what should be a very interesting two days of data.

Wednesday:

• Forget FICO: how data is changing the rules of credit and underwriting 
• Kleiner Perkins’ Michael Abbott: It takes two (teams) to build a successful app
• Without human input augmentation, algorithms alone are making us dumber
• SDN can turn the network into a big data “curator,” claims Juniper
• It’s not Skynet yet: In machine learning there’s still a role for humans 
• Data science is not enough. We need data intelligence too
• If you think big data is big now, just wait for the internet of things
• EMC’s Paul Maritz: it takes leadership to move companies toward a data-driven future
• How Aetna uses patient data to prevent diabetes and heart attacks
• Even the CIA is struggling to deal with the volume of real-time social data
• How coding contests can be better at solving problems than Harvard
• For big data achievements, IT and analysts need to work together
• Big data is still hard, but it gets better
• From Amazon’s top data geek: data has got to be big — and reproducible
• Beyond the Like button: Putting social networks to work for us
• Do we need internet exchanges for public cloud data portability?
• People will give up their personal info if you give them a good reason
• Six ideas from entrepreneurs for solving your big-data problems

Thursday:

• Why Nuance sees the semantic web as a key to smarter natural language interfaces
• Big data analytics is great but it’s no a silver bullet
• It’s not enough to just have information — intelligence requires context
• Hadoop: “It’s damn hard to use”
• Want a bigger/greener/more agile data center? Use the data
• Getting beyond the cult of big data
• How search can solve big data problems
• Hadoop applications abound, but Hadoop still needs improvement
• If the future of BI is Hadoop, SQL and the cloud are the glue
• Why Guavus analyzes lots of telecommunications data before storing it all
• Pursuing big data utopia: What realtime interactive analytics could mean to you
• Nasdaq on the virtues of the public cloud
• No, not every database was created equal. Here’s how they stand out

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Why the big data startup boom will likely be short-lived
• The Red-Hot Data Warehouse Market: Who’s Buying Next?

A key question becomes how this shift will become a reality and where we will look for a blueprint to begin. I think the answer, or at least the opportunity to see further, comes from “standing on the shoulders of giants.” And in this case specifically, I’m talking about the consumer internet giants like Google, Facebook and Amazon.

These companies have created significant new business value and blazed new trails in developing ways to manage and extract meaning from massive amounts of data. As a result, they’re able to deliver meaningful products, features, and experiences rapidly to their customers — essentially,giving customers what they want, when they want it and where they want it. Wouldn’t it be nice for traditional enterprises to have the same capabilities?

The traditional enterprise must learn from internet technology

Powered by new data fabrics with custom-built infrastructure, these consumer internet companies interact and serve their customers in the context of who their customers are, where they are and what they are doing in the moment. They are building, deploying and scaling at an unprecedented pace. They are storing, managing and delivering value from large data sets, and they knit all of this together on one unified platform that supports their businesses.

Paul Maritz at Structure 2011
(c) Pinar Ozger

Now add to this mix the emergence of the “internet of things,” the fact that telemetry will become pervasive in coming years. Everything from a fridge to a jet engine will be dialing home in the future, constantly reporting its state. This will drive a new avalanche of data that will arrive in huge quantities and will need to be ingested and reacted to in real time.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Infrastructure Q3: OpenStack and flash step into the spotlight
• The fourth quarter of 2012 in cloud

Paul Maritz, the EMC chief strategist who leads the spin off, expects it to be a $1 billion business within 5 years, but also warned it will need to make $400 million in investments to get to that point.

EMC chairman and CEO Joe Tucci

Pivotal is 69 percent owned by EMC, 31 percent VMWare with about 1,250 employees and $300 million in revenue, Tucci said. As has been reported, EMC contributed Pivotal Labs, Greenplum and VMware ponied up Cloud Foundry, Spring and Cetas.

“It’s not the riskiest thing we’ve ever done with an experienced executive, Paul Maritz, taking charge,” Tucci said.

Structure 2011: Paul Maritz – CEO, VMware

Maritz is the former CEO of VMware, and was a longtime top executive at Microsoft. He will also speak next week at GigaOM’s Structure: Data event in New York City.

“We have a great opportunity to bring EMC and VMware and Pivotal together and create great value for customers who want to deal with a few strategic suppliers,” Tucci told analysts and press.  But he also emphasized choice and flexibility. “If VMware wants to do something in storage that EMC might not like, they can do it.”

What the newly minted  Pivotal Initiative brings to the table  is Greenplum’s parallel query and data processing strengths; Gemfire’s ability to rapidly ingest events (lots and lots of events);  Cloud Foundry’s application development and deployment strengths and Spring’s Java rapid application development framework. The latter “gives us integration with legacy data sources and applications,” Maritz said.

The Cloud Foundry PaaS, which was initially built and still runs on VMware’s vCloud technology will continue to do so but will also run on Amazon Web Services, Maritz said. That whole openness message was repeated throughout this morning’s marathon event.

VMware also used Wednesday’s event to outline its Hybrid vCloud services with which it hopes to strengthen its public Infrastructure as a Service play against Amazon.

This story was updated at 9:14 a.m. PST to include Paul Maritz’ remarks about the spinoff’s prospects and assets.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Cloud and data third-quarter 2012
• IT spending update, third quarter 2012

One can only hope that Gelsinger, EMC CEO Joe Tucci and EMC chief strategist Paul Maritz will put some clarity around all this Wednesday when they talk to institutional investors. This event has been billed as an update on the Pivotal Initiative, the aforementioned spin-off which was finally announced in December.

Meanwhile, as talk of VMware’s proposed AWS killer circulated, Forrester analyst James Staten had his own interesting take on how VMware should attack its public cloud problem and it doesn’t hew to VMware’s vSphere-and-vCloud Director blueprint. That’s a non-starter in this new world of mobile and net-new applications, he wrote:

“What you should be doing is admitting you screwed up with vCloud Director 1.0 and 1.5 and kicking ass in engineering to get a true cloud to market ASAP.Your real threat? CloudStack and OpenStack atop Xen or KVM sold to a new cloud administrator inside the enterprise who starts with the service elements of a cloud the DevOps crowd values and worries less about the underlying abstraction layers and infrastructure. This is your real threat.”

GigaOM Pro analyst Jo Maitland took those thoughts a step further. “VMware should buy Eucalyptus for customers that want an internal cloud that is compatible with AWS and build an OpenStack alternative to AWS just like everyone else. vSphere and vCloud Director do not a cloud make.”

AWS partners shouldn’t be shocked, shocked!

Some providers of AWS monitoring services expressed outrage last week after Amazon offered a free trial of its own Trusted Advisor service. The issue is that Trusted Advisor offers configuration advice and other perks that compete with some of what Newvem, Cloudyn, Cloudability and other third parties offer.

Here’s the thing: Trusted Advisor has been around for a while — GigaOM reported on it last June. As Ed Byrne, CEO of CloudVertical, another monitoring provider who seemed less perturbed by the move, put it: “It’s been in beta for probably six months — everyone in the industry knew it would exit beta and become part of the basic offering, It’s a little rich to cry wolf now.”

Indeed. Any third party who thinks that AWS won’t keep building more services and capabilities is nuts. It’s like Amazon started out constructing a big, airy room with lots of space and gaps to be filled by third parties but then starts sucking all the air out of that room as it adds more of its own stuff.

Byrne summed it up: “If we have to make our business hoping AWS won’t do obvious value-adds for their customers, we are dead in the water. Of the $1.7 trillion per year spent on IT infrastructure, probably $3 billion is on AWS. There’s a whole lot of people trying to understand cloud economics for their own organization, for a migration, for a private or hybrid strategy [there's] plenty of scope for providers like ourselves and the others.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Federated clouds: for when one cloud isn’t good enough
• Microsoft Azure: What It Is, What It Costs and Who Should Care
• The fourth quarter of 2012 in cloud

At that new outfit, Nielsen will report to Paul Maritz, who will head up that cloud computing/big data spinoff of EMC and VMware, according to a document filed with the Securities & Exchange  Commission.

The move, which according to the SEC document, was official as of December 5 but not widely known. But it comes as little surprise because Nielsen — who was most recently co-president of VMware’s application platform group – was heavily involved with  Cloud Foundry, one of the VMware technologies turned over to the new spinoff. And he and Maritz go way back — they worked together at Microsoft more than a decade ago.

Having said that, there is concern of a continued brain drain at VMware. CTO Steven Herrod, a widely respected exec, is leaving for a VC gig at General Catalyst Partners.

With Nielsen’s move, another piece of the Pivotal Initiative puzzle falls into place.  Personnel from VMware and EMC are shifting around to staff up this new effort which combines IP from both parent companies. Some GreenPlum big data know-how and Pivotal Labs agile development from EMC as well as Cloud Foundry PaaS, Cetas analytics, and vFabric from VMware are flowing into the spin off.  Mark Lucovsky, who helped spearhead Cloud Foundry, is staying at the VMware mothership.

EMC, VMware and Pivotal Initiative folks will talk more specifics about the cloud and big data spin off March 13 at a NY event, VMware CEO Pat Gelsinger said on Monday’s earnings call. 

Feature photo courtesy of Flickr user Luke Kilpatrick

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• Infrastructure Q2: Big data and PaaS gain more momentum
• Cloud and data third-quarter 2012