17 Comments

Summary:

Every so often a new technology emerges that changes everything. In the world of storage, the last major media shift was the move to hard disk drives (HDDs) from tape. While tape is still around today as a target for backups and archiving (it’s cheap, durable, […]

Hard disk on fireEvery so often a new technology emerges that changes everything. In the world of storage, the last major media shift was the move to hard disk drives (HDDs) from tape. While tape is still around today as a target for backups and archiving (it’s cheap, durable, and portable for offsite storage), disk owns the vast majority of primary storage.

Today we find ourselves on the cusp of a transition in storage as massive as the move from tape to hard disk drives — the move from hard disks to solid-state disks.

CAPEX, OPEX and IOPS

From a capital expense (CAPEX) perspective (specifically the cost of acquiring a unit of storage), disk was more expensive than tape when HDDs started the last storage media disruption. Tape was also arguably cheaper from an operating expense (OPEX) perspective (specifically the maintenance cost including power, space, and labor). But from a cost-of-input/output-operations-per-second (IOPS) perspective, there was no contest. Anything that required fast access to storage moved to disk in short order and a new set of applications was enabled. Cost per IOPS won.

While it may appear that the transition from HDDs to SSDs (also popularly known as flash memory) will mean higher CAPEX on a cost-per-unit-of-storage basis, that may be misleading. Amdahl’s Law helps us to understand the maximum improvement to a system when only part of the system is improved. Mainframes leveraged Amdahl’s Law to maintain balance among the various components (compute, storage, and so on) in order to optimize overall system performance. Throughout the evolution of open systems this sense of balance has been lost. While Moore’s Law has been applicable to storage in terms of cost per unit of storage, storage IOPS have not kept pace with compute IOPS. The system is out of balance. Solid-state disks will help us regain the balance between storage and compute, so CAPEX per IOPS may well decrease substantially.

OPEX will also decrease as SSDs replace hard disks. Power consumption is lower with SSDs than with hard disks, so electricity costs per IOPS will also drop. Solid-state disks will bring better performance, reducing the number of systems and IT professionals required to do the same job. And since SSDs have no moving parts costs related to maintenance should decrease.

Just as with the last storage discontinuity, total cost per IOPS will be much lower for SSDs relative to hard disks in many, if not most, cases. Consequently I believe that, in the coming 5-10 years, SSDs will replace HDDs en masse. Cost per IOPS will win again.

And just as the last transition enabled a new set of applications, so too will this one. Already consumer devices like laptops, iPods, and the iPhone have put SSDs to good use. This is just the tip of the iceberg. Here are a few opportunities that come to mind:

1. Enterprise storage.

The next EMC and NetApp will likely be built off the back of the move to SSDs. Incumbents will have difficulty changing their mental models to address the new world. Existing vendors are already introducing hybrid solutions into the market, which is exactly what you would expect of incumbent players. But their software and systems were designed with assumptions about spinning media that are not appropriate in a solid-state world. There is room for at least one massive, pure-play SSD enterprise storage vendor.

2. Enterprise applications.

The vast majority of today’s application software was also written with hard drives in mind. Just as with storage, there will be an opportunity for applications to be written from the ground up with solid-state disks as the sole target. Will a new type of database emerge? If you designed the perfect database today with solid state in mind, would it look like Oracle or MySQL? Will SSDs empower real-time analytics, supply chain management and financial reporting? How will an order of magnitude improvement in IOPS change the art of the possible in customer-facing applications?

3. Consumer applications.

The largest consumer Internet players spend vast sums of capital on in-memory solutions and on operating costs associated with massive farms of servers and hard disk drives. Delivering results to a search query, placing the right advertisement in front of the right consumer at the right time, running a massively multiplayer online game, and building personalized pages are IOPS-intensive applications operating on gargantuan data stores. SSDs will dramatically lower the cost of providing services like these, ushering in a wave of innovation. And hopefully this will empower a bunch of hackers with little capital to disrupt some very large businesses as well as to invent all new ones.

While the world is suffering from a nasty recession, innovation marches on. Solid-state drives will help us regain balance between compute and storage, which will lead to lower overall costs and usher in a new class of applications. I can’t wait.

test_9207*I would like to thank John Colgrove for reading a draft of this note and providing his feedback. John is an EIR at Sutter Hill Ventures, previously served as a Symantec Fellow and VP of Technology Strategy for the company’s Data Center Management group, and was a founding engineer of storage software leader VERITAS Software which was acquired by Symantec.

Mike Speiser is a Managing Director at Sutter Hill Ventures. His thoughts on technology, economics and entrepreneurship will appear at this time every week.

  1. James Salsman Sunday, May 31, 2009

    THANK GOODNESS!

    How many people have ever had to try to recover from a dead hard drive? I make backups, but that hasn’t shield me from grief by osmosis from my family and friends who aren’t as diligent.

    After this, only fans and keyboards are left to have eliminated moving parts.

    Share
  2. With 64 bit address spaces, why even have IOPS for persistent storage? Many design assumptions in hardware architecture, databases, operating systems, and user-facing representation of storage should be challenged with this change.

    Share
  3. I have home brewed a 64 gig dynamic ram disk drive that uses a PCI-e X1 slot in my computer.

    I use it for my swap drive, temp files and a cache of files on a LRU basis.

    People with massive hard drives still only use about 10% of their files as a common ‘working set’ of data. The rest just sits on the drive taking up space most of the time.

    I have seen people build gigantic storage into a NAS to store all their DVD movies to have them available for example.

    I handle that different and keep only the first 3 minutes of the flick on the hard drive. I have a home brew mod to a 300 cd changer that now feeds the discs in to 3 dvd drives like a robot and will pick and stream the movie to the NAS to buffer up much faster than the movie is viewed. By the time the first 3 minute header on the drive is viewed, there is enough of the selected dvd buffered to the disc that a switch to that file is then possible to continue the stream and in under 10 or 12 minutes the whole movie resides on the hard drives.

    Makes no sense to keep the whole thing there just taking up space ,since some movies may not be watched except once or twice a year.

    Share
  4. If I access the movie inventory list from my office computer to pick the movie to be watched in the home theater this evening at the end of the day, by the time I go get a cup of coffee and pop some popcorn for the family there is enough buffered to the NAS and transferred to the Media PC in the Home Theater that I can watch the pick of the evening since it is way ahead on the buffering of the flick and is fully resident on the media PC within 15 minutes of the initial pick being made. Even if I am quick with the popcorn probably half the movie is already there by the time we sit down to watch.

    Share
  5. seriously, once the prices drop and the internet is available everywhere we are going to see internet devices that are only feasible because of solid state. it is so exciting to think about. i have visuals and schematics in the brain for several of them. and all because of internet paired with hardware that has no moving parts and fast access. here is a youtube video of samsungs ssds demonstrating their speed: http://www.youtube.com/watch?v=96dWOEa4Djs
    future here we come!

    - jason nadaf

    Share
  6. Article FAIL.

    The failure to mention the limited write cycle issues just destroys the credibility of the issue. There are significant obstacles rather than just price alone.

    Share
    1. There is a well-known burnout issue with MLCs today (not so with SLCs, but they are expensive). I’m confident that those problems will be solved.

      Over the years there were many “issues” with Moore’s Law and CPUs. The incentives were significant enough that those problems were solved (and continue to be solved) by great engineering. I believe the same will happen here, as flash such a jump in terms of random reads, power, and reliability.

      Share
  7. Amdahl’s and Moore’s laws play in this, but don’t forget Amara’s Law. SSD’s will change the market, but not immediately and far beyond simply swapping drives for SSDs. I posted on this here: http://tinyurl.com/q4c2yn

    Share
    1. Fixed the broken link from my name…

      Share
    2. Also known as exponential growth ;–)

      The question is how far along are we already? I’m with you. This will take time, but the beauty of exponential growth is that when it hits, it hits with force…

      Share
  8. [...] Mike Speiser discussed recently, flash solid-state drives (SSD) will enable a once-in-a-decade improvement in storage [...]

    Share
  9. [...] Mike Speiser discussed recently, flash solid-state drives (SSD) will enable a once-in-a-decade improvement in storage [...]

    Share

Comments have been disabled for this post