How to bring eBay’s infrastructure to every company

Earlier this month, Hugh E. Williams made the move from vice president of experience, search and platforms at eBay to senior vice president of research and development at Pivotal, the cloud computing and big data vendor that spun out of EMC and VMware in 2012. The move was officially announced on Wednesday, and he came on the Structure Show podcast to explain why, and how he expects his experience operating advanced infrastructure at web companies will help him lead the development of products that can make life easier for all types of companies.

Here are highlights of that talk, but you’ll want to listen to the whole thing for more details on specific use cases for big data and why he thinks Pivotal’s technology portfolio, in particular, is well suited for enterprise deployments.

Download this episode

Subscribe in iTunes

The Structure Show RSS Feed

Showing the world what’s possible

“I’ve seen from the inside the power of using sort of a data lake to really change a business and really put that business on a really fast, iterative trajectory,” Williams said, commenting on his tenure at eBay and Microsoft(s msft) before that.

The type of iterative business he’s talking about is able to use data to build products and services they think customers will want, then analyze the data after implementing it to see if it worked. It’s a closed-cycle environment that’s made possible when companies have a solid grasp on everything from server infrastructure up to data science.

“You go outside those top internet companies, and companies don’t have that infrastructure, they don’t have that way of working, they don’t have that knowledge — and it’s pretty clear to me that’s what the world needs,” Williams said. “… [Companies in all sorts of industries] are sitting on vast amounts of data.”

Hugh E. Williams, Pivotal's SVP of Research & Development
Hugh E. Williams, Pivotal’s SVP of Research & Development

Recognizing the opportunity — and being able to act

“I think most of the customers that I’ve spoken to, and folks I’ve spoken to … at the very least sense that this is where the world’s headed. And I think there’s varying levels of maturity of that,”  he said. “… People get that they’ve got a lot of data, but what I think they need help with is maybe infrastructure to store it, the technology to process it, and ultimately the DNA in their organization to really move in that fast iterative circle.”

He cited an airline industry company that has data on everything from maintenance data to weather data to data about when customers check in for their flights, and it’s just now figuring out how to use it all.

Where Pivotal has work to do on its technology

However, as much as Williams believes companies are ready to adopt big data technologies, he also acknowledges that vendors — even his new employer — must keep working to make them easier.

“First of all, we need to continue to invest in building world-class data lake infrastructure, for want of a better phrase,” he explained. “If you’ve used Hadoop, if you’ve tried to deploy Hadoop into an organization, you know that’s a pretty difficult proposition. It’s pretty brittle and it’s missing a lot of the features and stability that you’d expect from an enterprise-grade application. … There’s a big piece of R&D there.”

DSC_4682 - HR-small
A big Hadoop cluster at Williams’s former employer, eBay. Source: eBay

On top of that, vendors need to make it easier for customers to build new types of applications — something Pivotal should have as a priority considering that its name comes from Pivotal Labs, the application-development specialist EMC acquired in 2012.

“The data piece is foundational, and it’s a foundation on which you build applications,” Williams said. “They could be applications that applied scientists want to use, that a finance team wants to use, and they might be applications on which you build customer facing applications — which is typically what the large internet giants are doing.”

And then there’s the cloud. “If you want to have a flexible, world-class, enterprise-grade Hadoop infrastructure, then that’s gotta really run as a platform that runs as a service,” he said. “… The partnership between Cloud Foundry and the Hadoop work that we do needs to be very, very tight and deep if you’re going to realize that vision.”

Big data sells, but who’s buying?

If Williams is right, vendors like Pivotal can play a role not just in serving large enterprises and other mainstream companies, but even up-and-coming web companies that are trying to scale up their infrastructure to meet growing user bases.

“I think the really big players — the largest of the large internet giants — are always gonna want to do this themselves because they have the resources to want to go do that, and they want to build the infrastructure in a very specialized way for the applications they’ve got,” he acknowledged. “But I think once you move beyond the top two or three, I think there’s a huge opportunity there.”

structuredata2014_300x200_editpost2We’ll hear firsthand what Williams’s new boss, Pivotal CEO Paul Maritz, thinks about his company’s strengths, weaknesses and promise in the new data economy in a couple months. Maritz will be sitting down with our own Om Malik on March 19 for a fireside chat at our Structure Data conference in New York.