Blog Post

How to bring eBay’s infrastructure to every company

Earlier this month, Hugh E. Williams made the move from vice president of experience, search and platforms at eBay to senior vice president of research and development at Pivotal, the cloud computing and big data vendor that spun out of EMC and VMware in 2012. The move was officially announced on Wednesday, and he came on the Structure Show podcast to explain why, and how he expects his experience operating advanced infrastructure at web companies will help him lead the development of products that can make life easier for all types of companies.

Here are highlights of that talk, but you’ll want to listen to the whole thing for more details on specific use cases for big data and why he thinks Pivotal’s technology portfolio, in particular, is well suited for enterprise deployments.

Download this episode

Subscribe in iTunes

The Structure Show RSS Feed

Showing the world what’s possible

“I’ve seen from the inside the power of using sort of a data lake to really change a business and really put that business on a really fast, iterative trajectory,” Williams said, commenting on his tenure at eBay and Microsoft(s msft) before that.

The type of iterative business he’s talking about is able to use data to build products and services they think customers will want, then analyze the data after implementing it to see if it worked. It’s a closed-cycle environment that’s made possible when companies have a solid grasp on everything from server infrastructure up to data science.

“You go outside those top internet companies, and companies don’t have that infrastructure, they don’t have that way of working, they don’t have that knowledge — and it’s pretty clear to me that’s what the world needs,” Williams said. “… [Companies in all sorts of industries] are sitting on vast amounts of data.”

Hugh E. Williams, Pivotal's SVP of Research & Development
Hugh E. Williams, Pivotal’s SVP of Research & Development

Recognizing the opportunity — and being able to act

“I think most of the customers that I’ve spoken to, and folks I’ve spoken to … at the very least sense that this is where the world’s headed. And I think there’s varying levels of maturity of that,”  he said. “… People get that they’ve got a lot of data, but what I think they need help with is maybe infrastructure to store it, the technology to process it, and ultimately the DNA in their organization to really move in that fast iterative circle.”

He cited an airline industry company that has data on everything from maintenance data to weather data to data about when customers check in for their flights, and it’s just now figuring out how to use it all.

Where Pivotal has work to do on its technology

However, as much as Williams believes companies are ready to adopt big data technologies, he also acknowledges that vendors — even his new employer — must keep working to make them easier.

“First of all, we need to continue to invest in building world-class data lake infrastructure, for want of a better phrase,” he explained. “If you’ve used Hadoop, if you’ve tried to deploy Hadoop into an organization, you know that’s a pretty difficult proposition. It’s pretty brittle and it’s missing a lot of the features and stability that you’d expect from an enterprise-grade application. … There’s a big piece of R&D there.”

DSC_4682 - HR-small
A big Hadoop cluster at Williams’s former employer, eBay. Source: eBay

On top of that, vendors need to make it easier for customers to build new types of applications — something Pivotal should have as a priority considering that its name comes from Pivotal Labs, the application-development specialist EMC acquired in 2012.

“The data piece is foundational, and it’s a foundation on which you build applications,” Williams said. “They could be applications that applied scientists want to use, that a finance team wants to use, and they might be applications on which you build customer facing applications — which is typically what the large internet giants are doing.”

And then there’s the cloud. “If you want to have a flexible, world-class, enterprise-grade Hadoop infrastructure, then that’s gotta really run as a platform that runs as a service,” he said. “… The partnership between Cloud Foundry and the Hadoop work that we do needs to be very, very tight and deep if you’re going to realize that vision.”

Big data sells, but who’s buying?

If Williams is right, vendors like Pivotal can play a role not just in serving large enterprises and other mainstream companies, but even up-and-coming web companies that are trying to scale up their infrastructure to meet growing user bases.

“I think the really big players — the largest of the large internet giants — are always gonna want to do this themselves because they have the resources to want to go do that, and they want to build the infrastructure in a very specialized way for the applications they’ve got,” he acknowledged. “But I think once you move beyond the top two or three, I think there’s a huge opportunity there.”

structuredata2014_300x200_editpost2We’ll hear firsthand what Williams’s new boss, Pivotal CEO Paul Maritz, thinks about his company’s strengths, weaknesses and promise in the new data economy in a couple months. Maritz will be sitting down with our own Om Malik on March 19 for a fireside chat at our Structure Data conference in New York.

4 Responses to “How to bring eBay’s infrastructure to every company”

  1. Philip Charles Cohen

    Did someone mention eBay/PayPal?

    Shill bidding fraud by professional sellers on eBay nominal-start auctions is demonstrably endemic; bidders who bid early and often on such auctions will most likely be sellers’ shills, and eBay Inc. is – also demonstrably – the greatest knowing and calculated facilitator of such fraud on consumers that the world is ever likely to know and, doubtless, there will be a trickle down effect of like criminal activity to other eBay operations …

    The only thing that surprises me about this utterly unscrupulous corporation is that it has not yet summoned up the gall to further obscure the endemic shill bidding fraud by converting its auctions wholly to its devious, seller-selected, “private listing” format, or by changing bidding IDs to “Bidder 1”, “Bidder 2”, “Bidder 3”, etc, as they initially were going to do/did when Donahoe first introduced the additional layer of anonymity for (shill) bidders in 2008, nor – surprisingly – have they yet ceased to publish the details of “completed” listings, a post-sale analysis of which, with some effort, can clearly demonstrate this endemic fraud and eBay’s calculated facilitation thereof …

    And what does the “smart money” on Wall Street think about eBay? Well, in August 2007, when the “Pain From Bain”, Johnny Ho, was already effectively in control of eBay, the share prices of both eBay and Amazon were ~$40; today, eBay is ~$53; Amazon is ~$400 … Now, seriously, what more can be said, other than …

    eBay / PayPal / Donahoe: Dead Men Walking …

    I also have to wonder if Pierre Omidyar has ever thought about just how much more fabulously wealthy he would now have been had he, before Johnny Ho took control, traded his eBay stock in for Amazon stock, or had he not ok’d the handing over of the control of eBay to this incompetent, destructive, unscrupulous, narcissistic sociopath?

  2. Philip Charles Cohen

    “Williams advised sellers who want to be successful with eBay search that they 1) need to be found in search; 2) have someone click on their item and 3) sell that item. …” – Hugh Williams (Jun 2013)
    [Is it any wonder that Hugh is now “retired” from eBay “to spend more time with his family” …]

  3. ok just please do not bring anything of eBay’s service culture and customer orientation to the new company … technological innovation has unfortunately destroyed traditional customer service at eBay … in my opinion