Blog Post

Four tips to take you beyond the big data hype cycle

Stay on Top of Enterprise Technology Trends

Get updates impacting your industry from our GigaOm Research Community
Join the Community!

As 2013 kicked off Gartner analyst Svetlana Sicular noted in her blog that big data is sliding down into the Trough of Disillusionment, a steep cliff in the Gartner Hype Cycle that follows the Peak of Inflated Expectations. (If you’re not familiar with the Gartner hype cycle, check out the illustration on Svetlana’s blog.)

In my experience with big data, there’s no reason for disillusionment. Big data analysis can create huge amounts of value. As with most worthwhile pursuits, it takes work to unlock that value. In the last three years, as a member of the CIO staff at Intel, I’ve spent a big chunk of my time developing business intelligence and analytics solutions that have resulted in tremendous cost and time savings and substantially improved time to market.

Beyond my own personal anecdotes, Gartner’s most recent Hype Cycle report seems to agree that there is in fact substance behind the hype: if you can stick it out past knowledge gathering and initial investment to actual deployment, you’ll move beyond disillusionment and start seeing results. As a matter of fact, many organizations are already finding the value in Big Data and investing even more heavily in related projects for 2014.

However, the report also notes that 2013 is the year of experimentation and early deployment, which is why many may not be singing the praises of big data initiatives just yet.

If you find yourself in this stage, there’s no reason to despair. Here are four tips for steering clear of the ‘trough of disillusionment’ and deriving value from your big data implementation.


Think even bigger.Think of a larger, more comprehensive model of business activity and figure out how you can populate it from as many data sources as possible. Then you can see the big picture. After you envision what infrastructure you need to support data at that scale, ask yourself if you could increase your data by a factor of 10 or more and still use the same infrastructure.

This is what Oregon Health & Science University (OHSU) is doing on a big data project to speed up analysis of human genomic profiles, which could help with creating personalized treatments for cancer as well as supporting many other types of scientific breakthroughs. Calculating about a terabyte of data per patient, multiplied by potentially millions, OHSU and its technology partners are developing infrastructure to handle the massive amount of data involved in sequencing an individual’s human genome and noting changes over time. With breakthroughs in big data processing, the cost for such sequencing could come down to as low as $1,000 per person for this once-elite research, which means demand will skyrocket. And when demand skyrockets, so will the data.

Find relevant data for the business. Learn from line of business leaders what their challenges are, what’s important to them, and what they need to know to increase their business impact. Then search for data to see if you can help them solve their business problems. That’s exactly what happened with Intel’s internal big data initiatives. We were asked to help the sales team focus on which resellers to engage, when, and with what products. In 2012, the results of this project drove an estimated $20 million in new revenue and opportunities, with more expected in 2013.

Be flexible. We are in a phase of rapid innovation. It isn’t like implementing enterprise resource planning. From a technology standpoint, you must be fluid, flexible, and ready to move to a different solution if the need arises. For example, the database architecture built to collect “smart grid” energy data in Austin, Texas, with Pecan Street Inc., a nonprofit group of universities, technology companies, and utility providers, is now on its third iteration.

Austin's 's Pecan Street Project. Photo courtesy of Pecan Street Inc.
Austin’s ‘s Pecan Street Project. Photo courtesy of Pecan Street Inc.

As smart meters generate more and more detailed data, Pecan Street Inc. is finding new ways for consumers to reduce energy consumption as well as helping utilities better manage their grids. But Pecan Street had to be willing to keep changing its infrastructure to meet demand. The bottom line: If you think you know what tools you need to build big data solutions, a year from now it will look different. Be ready to adapt.

Connect the dots. At Intel, we realized there could be tremendous benefit in correlating design data with manufacturing data. A big part of our development cycle is “test, reengineer, test, reengineer.” There is value in speeding up that cycle. The analytics team began looking at the manufacturing data—from the specific units that were coming out of manufacturing—and tying it back to the design process.

Epitaxial wafer with reflector
Epitaxial wafer with reflector

In doing so, it became evident that standard testing processes could be streamlined without negatively impacting quality. We used predictive analytics to streamline the chip design validation and debug process by 25 percent and to compress processor test times. In making processor test times more efficient, we avoided $3 million in costs in 2012 on the testing of one line of Intel Core processors. Extending this solution into 2014 is expected to result in a reduced expenditure of $30 million.

We are only at the beginning of understanding how we can use big data for big gains. Far from being disillusioned with big data, we find many exciting possibilities as we look at large business problems holistically and see ways to help both the top line and the bottom line, all while helping our IT infrastructure run more efficiently and securely. It’s not easy to get started, but it is certainly well worth the time and effort.

Ron Kasabian is general manager of big data solutions for Intel’s Data Center & Connected Systems Group.

8 Responses to “Four tips to take you beyond the big data hype cycle”

  1. geckorion

    Big Data can be useful, but only if companies have the right pieces and plan in place beforehand. As many of the commenters below mentioned, having a Big Data program in place is not a guarantee of increased revenue. It is more about realistic expectations about what you are looking to accomplish and whether you have the internal communications, skills, and bandwidth to fully use the data you are gathering.

    I wanted to share a video that I think can be helpful for your readers that deals with planning and executing a Big Data program. ( This video is based off of TEKsystems research and delivers the message in a cute way through multiple sci-fi references.

  2. Alexandra Gillies

    Possibly a good thing the hype is subsiding – as it does obfuscate! Building on the comments from Michael below, perhaps two tips the article might have included to avoid and/or mitigate disillusionment, is (1) the importance of ensuring appropriately skilled/knowledgable people are given the space, time and support to focus on the questions, data, method and interpretation – ultimately leading to the extraction of knowledge/value/intellegence i.e. people and how the are organised is the real locus of value; and (2) managing expectations on time – it takes years to build competency in this space — and outsourcing to expedite can be fraught with complexity and risk.

  3. Ralph Winters

    This is not necessarily a bad thing. All technologies go through cycles. Hopefully, better products arise because of it. Big Data was/is not presented according to any standard criteria that I know of, and as a result customers are confused. Maybe the vendors can take it back to the drawing board and retool their offerings.

  4. Data 101

    Ron, very good perspective on Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at

  5. It is important to remember that data is not knowledge…it is easy to generate data but it is extremely difficult to 1) identify what the correct question is that needs to be answered, then 2) identify the data that is necessary to address the question and finally, 3) determine and implement the appropriate methods to analyze the data to provide a solution to the real question…this is a major problem in medicine where the lure of big data may not adequately address the real problem in medicine which is getting the diagnosis correct to start with…you cannot associate all the genomic data with medical improvements if you are dealing with syndromes and complex disorders and not “simple diseases”

    • Geoffrey West of the Sante Fe Institute recently wrote in Scientific America (see May 2013: Wisdom in Number) that: “The trouble is, we don’t have a unified conceptual framework for addressing questions of complexity. We don’t know what kind of data we need, nor how much, or what critical questions we should be asking. ‘Big data’ without ‘big theory’ to go with it loses much of its potency and usefulness, potentially generating new unintended consequences.”

  6. We wrote up something similar a few months back. While there’s definitely a hype cycle for any new concept, the one for Big Data is different because the place on the cycle depends very much on the perspective of the big data user.

    There are companies doing amazing things that reject the term “big data” and others that are spending money on applications that aren’t delivering to expectations. I’d say that Big Data is on every point of the hype cycle simultaneously, mostly because the term itself is ambiguous and used differently by many.

    • Anyone thinking big data will be a great success just need to look at Yahoo. They have been using Hadoop for a long time. Now have they made any killer profit yet? If they who know Hadoop inside out could not earn much from it after so many years what makes anyone else still buy into the endless hype then?

      You can only be gullible for so long some times.