If you wanted an image to define success in the technology industry, it would be tough to beat the hockey stick.
The proverbial “hockey stick demand curve” is the visual representation of ambition fulfilled. For years, scientists, marketers and others toiled on the left side of the chart trying to overcome technical challenges and understand market demand. Early adopters make the middle of the curve perk up and then, boom, the world discovers you. Sales skyrocket, revenue (and sometimes profits) zoom, competitors rush into the market and the virtuous cycle of lower prices, higher volumes, improved performance and greater variety begins.
After a few years, demand begins to taper off and the hockey stick starts to look like a gradual upward slope.
As sexy as the hockey stick curve is, it’s not the only for new industries or technology. The development and increase of the digital universe is creating a potential hockey stick for a very old technology — that of digital storage. This is an industry that has been around for over 56 years, but we’ll hit that hockey stick curve thanks to all of the digital content we’re sharing and creating. We will generate, replicate and consume 3.5 zettabytes (that’s 3.5 with 21 zeros behind it) of data in 2013 and the amount of digital data generated will grow by 42 percent a year. By 2020, the annual growth of the digital universe will come to 40 zettabytes.
Put another way, more than 5,200 gigabytes of data for every person on the planet in less than eight years.
A little history is in order
The digital storage industry experienced its first boom in the late 1950s and early 1960s when digital storage systems like IBM’s RAMAC, the first hard drive-based computer, debuted. Customers flocked to digital as a way to manage paper. (Ironically, a paper company, Crown Zellerbach, owned the first one.)
The third wave of demand is being driven by what you could call the digitization of daily life. Movies, phone communication, social networking, business processes and other activities are all shifting to digital. Latin America, Asia and Africa, meanwhile, represent five billion new consumers and businesses.
The Internet of Things—the coming network will be populated by connected washing machines, thermostats, sprinkler systems, household appliances and industrial equipment, which will mean another 150 billion connected devices generating data.
While this looks promising for the storage industry, it also creates a major challenge. Namely, how do we produce enough storage devices to hold all of this data? Through 2014, expected supply and demand levels for hard drives and flash are still in synch. Soon after, demand accelerates. By 2020, annual demand for new storage technology will hit 9.8 zettabytes; substantially less than the 40 zettabytes of data expected to be generated that year.
A coming shortage of storage
Like the Southeast Asian floods of 2011 showed, shortfalls are ultimately bad for everyone. So how is the world going to get around the problem? Several possibilities exist:
Don’t save everything. We don’t need all of the data, some will argue. Intelligent light bulbs with motion sensors will be able to track your movements during the day, but most of the information can be discarded or summarized. While this solution sounds attractive, this is likely not going to happen as getting rid of data would undercut the promise of Big Data. We don’t know what useful patterns will emerge unless we collect the data in the first place.
Build more factory capacity. This will likely happen, but it can’t make up the gap entirely. Investors and manufacturers are necessarily cautious when it comes to expanding capacity. According to Gartner, the price of a NAND flash fabrication facility will rise from $1 billion in 2004 to $18 billion by 2020. A new facility for producing hard drives costs about $35 to $50 million.
Recycle existing machines and systems. Possible, but the capacity and density of storage devices has grown faster than Moore’s Law. That means those old storage devices are much larger, consume more energy, hold less data and take up more room. In 2004, Sony excited tech enthusiasts with a 1 terabyte home server that cost $5,000. Now, notebook drives can support a terabyte.
Go back to tape and optical. Facebook is experimenting with caching data on Blu-ray, but latency and speed may keep a lid on how much businesses can take advantage of these kinds of technologies.
Technology to increase density. Several new and promising technologies are already in the works: NAND manufacturers are experimenting with 3D transistors while hard drive manufacturers will soon release disks based on heat-assisted magnetic recording.
Ultimately, we will see a combination of these solutions. It’s an opportunity that’s too big to ignore.
Albert “Rocky” Pimentel is executive vice president of Worldwide Sales and Marketing at Seagate.