Blog Post

Did Yahoo sow the seeds of its own demise with Hadoop?

Yahoo (s yhoo) is famous for saying that Hadoop, the data-storage-and-processing platform it helped develop into a big data juggernaut, is “behind every click” on the site. But as the world once again starts analyzing Yahoo’s myriad woes after Sunday morning’s ouster of embattled CEO Scott Thompson, I’m left wondering if its investment in Hadoop didn’t aid in the company’s demise — even if it’s a way down the long list of Yahoo’s mistakes.

While Hadoop might have given Yahoo a competitive advantage early on by letting it better analyze user behavior, it also leveled the playing field over time. For years, Yahoo has used Hadoop to better personalize the content its visitors see, both ads and articles, and now so does everybody else. In the meantime, however, Yahoo failed to evolve its business at the same pace as it did its analytic capabilities.

Big data is great, but it’s not a business model in and of itself. Yahoo might as well have just told its competitors, “We still suck, but we spent a lot building a great technology to help you compete.”

The company that has spent the last couple months suing Facebook over patents — even threatening some core components of the Facebook’s infrastructure — used to be on the forefront of open source technologies for webscale operations. Hadoop especially, which Doug Cutting created while a Yahoo employee and then released to the Apache Software Foundation, has always been a point of pride at Yahoo. The company used to maintain its own open source distribution of the platform, and even still ex-Yahoo employees now with Hadoop startup Hortonworks cite the overwhelming majority of the project’s code contributed by Yahoo employees.

But whatever benefits Hadoop brought to Yahoo in terms of letting it better targeting the right content to the right visitors, it also brought them to everyone else. If you talk to a company building an ad-targeting or web analytics technology, chances are Hadoop is somewhere in its technology stack. The aforementioned Hortonworks, which spun out of Yahoo last June, is now peddling Hadoop software and services to the world. A new startup called InsightsOne, also comprised of former Yahoo data experts, is selling a technology for helping marketers reach consumers wherever they happen to be on the web.

The big problem for Yahoo is that, increasingly, users and advertisers want to be everywhere on the web but at Yahoo. Maybe that’s because everyone else that’s benefiting from Hadoop, either directly or indirectly, is able to provide a better experience for consumers and advertisers alike. Facebook has been particularly adept at using and developing Hadoop to analyze behavior, and — despite potential privacy problems on the horizon — it’s eating Yahoo’s lunch in terms of attracting advertisers and eyeballs.

Not that Yahoo should be blamed for anything it did with regard to Hadoop; rather, it should be commended. It has changed the face of data management on the web like no technology — whether from Yahoo, Google, Facebook or whomever — likely will for some time. But I can’t think of Yahoo and Hadoop without thinking about Sun Microsystems, Java and MySQL, and we all know how that story ended.

8 Responses to “Did Yahoo sow the seeds of its own demise with Hadoop?”

  1. Andrew Newman

    Remember that Hadoop was open sourced around the time Microsoft was going to buy it the first time? A great idea if Yahoo became a Microsoft shop and you wanted to keep using it.

    Hadoop is taught at Unis and there are thousands of projects built around it – it’d be a great way to leverage if you were a tech company – see cool projects and reuse or buy. But since when was Yahoo a tech company?

    Technology is not why Yahoo is failing.

  2. Yuvamani

    If Yahoo did not do Hadoop, some one else would have done it. Take the NoSQL databases as an example, There was a clear need for a Dynamo from Amazon or Big Table from Google. And so somebody did MongoDB, Somebody did Redis, Somebody did couchdb. And yes Facebook open sourced Cassandra, But ended up using HBase from Hadoop.

    There was a clear need for a good map reduce system and somebody did it. To Yahoos credit they did the best and most popular one. I do not think the infrastructure itself is a competitive advantage in web land anymore although companies *have to* necessarily spend a lot of money and engineering on scaling their web infra !

    Take Google+ vs Facebook. All of googles infra and all of googles data centers does not make any difference in the social war.

    • Derrick Harris

      I agree and disagree. I think infrastructure is a competitive advantage, if only a small one, re: saving money and improving performance. I think Hadoop is actually more strategic, in that it really can let you display content in more effective ways or otherwise use data to your advantage.

      The key, though, is to improve the core business while also developing these complementary technologies. Facebook and Google do that well. Otherwise, as with many NoSQL projects (or Cloudera or Hortonworks, which Yahoo has a stake in), make the technology the business.

  3. Amirhossein

    It’s been said that information technology in general cannot bring competitive advantage since it is not sustainable. The same is true for Hadoop. I cannot judge if the open source nature of Hadoop make the sistainability issue worse.

    • Derrick Harris

      IT doesn’t generally give a sustainable advantage, I don’t think, but it can give a temporary one to early adopters. Even if Yahoo did everything right re: Hadoop, though, it never improved the core Yahoo platform simultaneously. When new competition comes along, there’s a vastly improved Hadoop thanks to Yahoo, but not a vastly improved Yahoo.

  4. I really hope that yahoo makes some significant changes. Yahoo perhaps is the only company that could actually become bigger than google if they make the correct decsions. I think that people have grown to dislike google so much that they would be willing to give yahoo a second chance.