42 Comments

Summary:

When last year’s hard drive shortage threatened Backblaze’s all-you-can-store cloud backup service, the company had to get creative to keep up its 50TB-a-day hard drive habit. The solution: external hard drives from retail stores and an army of volunteers making sure they kept coming.

hard drives
photo: Backblaze

“We buy lots and lots of hard drives . . . . [They] are the single biggest cost in the entire company.”

Those are the words of Backblaze Founder and CEO Gleb Budman, whose company offers unlimited cloud backup for just $5 a month, and fills 50TB worth of new storage a day in its custom-built, open source pod architecture. So one might imagine the cloud storage startup was pretty upset when flooding in Thailand caused a global shortage on internal hard drives last year.

Gleb Budman

“Literally overnight,” Budman told me, “… all the places we would go to get drives said, ‘Sorry, we don’t have any drives.'”

People assumed it was just a blip, and while Backblaze watched cautiously in the beginning, it figured it had enough hard drives stockpiled to make it through. However, when months passed and the situation only got worse — some suppliers were offering 3TB drives that used to cost $129 for around $600 — Backblaze knew it had to act. If the company didn’t want to change its pricing model or throttle users’ capacity, something had to give.

“That’s an absolute, just last, last, last resort,” Budman said.

It’s good to be a startup

Sometimes, it’s nice to have the flexibility of a startup. Rather than jack up prices or lower its revenue guidance, Backblaze just kept going about its business. Well, publicly, at least — behind the scenes, the company was working like crazy farming hard drives from the only places it could still get them at a reasonable price.

Its solution was to eschew the internal hard drives generally put inside servers and buy up the external hard drives sold for consumer backup at stores such as Best Buy, Fry’s and Costco. They fit nicely into Backblaze’s storage pods once removed from their protective enclosures, and the best part was that the 3TB drives Backblaze requires only cost around $169 apiece even during the height of the shortage. The company that builds Backblaze’s pods was even willing to “shuck” the drives for a couple bucks apiece, saving Backblaze a lot of manual labor in order to make its newfound source of capacity production-ready.

And then it happened: The shopping carts Backblaze was initially filling up gave way to two-hard-drives-per-person limits, which meant the company had to scramble. Keep in mind, adding 50TB a day means getting your hands on at least 14 new drives per day. In the end, Budman tells me, its ingenuity meant Backblaze was able to procure about 1,800 3TB hard drives — or about 5.5 petabytes worth of capacity — in the three months it was actively farming them.

Backblaze details much the process in a Tuesday-morning blog post, including the hijinks that followed as the company got creative trying to figure out ways around the new hard drive limits. Maps were drawn, employees were cut off from purchasing hard drives at Costco — both in-person throughout Silicon Valley and online (despite some great efforts to avoid detection, such as paying for hard drives online using gift cards) — and friends and family across the country were conscripted into a hard-drive-buying army.

A chart of local stores and employee purchases

What doesn’t kill you …

Budman said the shortage should have taught everyone a valuable lesson that hard drives aren’t commodities that will always be available at commodity prices, so you need to take advantage of low prices when you can. In fact, Budman said, one of the reasons he resisted venture capital initially was to help stave off the mindset that money from VCs is easy and companies don’t need to spend wisely.

It appears the strategy worked. Backblaze closed a $5 million funding round in July 2012, but it hasn’t abandoned its hard-drive-farming ways. Internal hard drive prices have stabilized, but even today if prices on external drives drop to less than those of internal drives (after factoring in the voided warranty and the cost to shuck them), Backblaze will snatch them up.

But it’s nothing like the glory days in late 2011 and early 2012. Here’s a video of Backblaze employee friend and NextPunch founder Vladik Rikhter telling his story of playing cat-and-mouse with Costco’s e-commerce system and ending up with “a Kia’s worth of drives” in his entryway.

  1. Neil Hinrichsen Tuesday, October 9, 2012

    In this day and age, a startup not using AWS?!

    1. I presume that’s an essential factor in being able to offer the prices they do? I know my 100GB or so of AWS storage costs me a chunk of change each month, and I guess that add’s up when you’re talking 50TB/day. Don’t forget that data accumulates. 50TB added each day is 1500TB after one month, and 18,000TB after a year. That’d cost over $1million on AWS, versus slightly less for physical disks (but, as time goes on, you’ll get returns on owning the physical disks vs renting space at Amazon).

      1. You have to keep in mind that hard drives have a finite life so you have to replace dying hard drives as well. Its not a buy once either way. You either rent space on Amazon or you are “renting” your physical hard drive at the cost of $169 for 2-3 years…

      2. @ SpecialK777

        If it’s just backups, data is stored most likely for a long time. That 3-5 years you mention could mean writing data just once. I’ll admit that I don’t know exact business model used by Backblaze, but “unlimited cloud backup for just $5 a month”. sounds like something that would make me just store the data and forget about it. Sure HDD not being used for 3-5 years might not function, but hopefully they do some form of duplication anyway.

    2. … Backblaze is a cloud backup service that sells space similar to AWS. Ex: If you were selling ice cream you wouldn’t rent time at DQ to make your customers’ cones. So, not using AWS because they are a separate backup service who manages their own uptime.

    3. Backblaze sells cloud backup service to customers, in my opinion, it would be très weird to purchase infrastructure from a company offering competing services.

    4. You should click on the link high up in the post referencing Backblaze’s architecture. It goes into detail on how it builds 135TB “storage pods” on the cheap.

    5. Their architecture, all up factoring in depreciation and management, costs about ten percent of storage at AWS.

    6. AWS is way more expensive.

      Backblaze’s pods cost $7,500 for 135TB or $55 per TB.
      AWS’s cheapest storage is about $35 per *month* per TB.

      Let’s be generous and say one employee needs to maintain a Pod once per month for one hour (way higher than our server management). Let’s assume they consume 1KW of power each. Even if the pod burned out every other year, unlikely but we’ll be generous, to Amazon that works out to ($3750) + (12 months*$100/hr) + ($0.10/KWh * 24hr * 365 days) = $5826/12 = $485 per month / 135 TB = $3.59 per TB per month. That’s assuming a 50% failure rate in HDDs per year.

      Let’s also not forget bandwidth. Backblaze is looking to upgrade to 10gbps for their datacenter (OC-192). Which runs about $200k per month from what limited data I could find. They currently have 40Petabytes which works out to $200k/40kTB = $5 per month per TB.

      So our total is about $9TB per month for Backblaze
      vs
      $30TB per month for AWS.

      But that’s just my estimate. You could easily cut almost all of those support time costs. You could very easily get 4 years not 2 years out of a pod and you could probably use green drives that use less than 200watts cutting your power by 80%. In other words $9TB is generous and assuming that they actually need 10gbps right now.

      1. AWS sells storage for $0.01/GB/month with Amazon Glacier. It’s S3 that is more expensive.

      2. Mind you that AWS’s storage costs include redundancy for the data, and the raw price per TB for Backblaze does not. You will probably need to multiply by 3-4x to get suitable redundancy.

    7. Why do so many people assume that AWS is the best/only option? Thank god Larry and Sergey didn’t stop at thinking Yahoo was the best we could ever do.

      1. Though, they DID want to sell to Yahoo early.

    8. What bit of Backblaze “offers unlimited cloud backup” do you not understand? They’re not a typical start-up, they’re effectively a 100% competitor to AWS in many ways! That would be like telling Oracle to buy their database expertise from Microsoft when faced with a DB consultant hiring shortage. Please!

  2. Zipperbits: I’ve seen it done. Example: http://s3rsync.com (no I do not work for them… I’ve considered paying for their service, but ultimately did not.)

  3. Yes hard drives can fail, but I have a 80GB drive in one of my 8 year old computers that is still going strong. It can be far cheaper to own the drives than to rent space on AWS.

  4. Something does not feel right about Backblaze bragging so much about their “drive farming’ exploits. While I agree that it was an innovative way to keep to their business goals, they did so by exploiting the retailers’ businesses. The retailers still made their money, but you could argue that it hurt their customer relationships. Alls fair and all that, but a company that boasts so much about something like this comes off as a bit tone deaf IMO.

  5. Psyphurr Lock Tuesday, October 9, 2012

    I have a small 4 drive 1TB network attached SAN at home thats been running strong for over 7 years. No drive replacements, notta. I keep the data replicated just in case of course. but its still running like it did on day one. So I agree, buying your own disks is certainly less expensive in the long run.

  6. Seems pretty clear who I’m NOT using for cloud storage/backup.

  7. @Not. AWS is the not best priced storage. There are other backup services. I personally use and love CrashPlan. They have multiple levels of scale ability. They support personal, SMB. and enterprise level of support. They support Windows, MacOS, Linux, even Solaris. I use the personal version at home. They have a ‘family’ plan that for $USD12/month you can backup up to 10 computers with unlimited data. This is handy when you have multiple laptops that may not be on your home network all of the time. There client can be used for both local backup also. You can also backup with other CrashPlan friend for free. All data is AES client side encrypted so they don’t see a thing.

    P.S. I know this sounds like an advertisement, but they really do a good job at cloud backup.

    1. If only they could handle the amount of files that I want to send them. I can’t even use it to backup one whole drive to another on a local system, let alone try to keep that in sync with their server.

  8. Mike McLaughlin Tuesday, October 9, 2012

    …..time and time again this is why small businesses fail……failing to plan. I know of no storage vendor that did not have a steady supply of HHDs this year. To find out that my accounting files, engineering drawings, contract were sitting on a Costco purchased aka sh$# drives I’d of sued these clowns at Backblaze. Folks checkout EMC’s Mozy, HP, hell I’d go to Dell before this moron……

    1. Mike,

      I can see your point to a degree, although I’d note two things: (1) Backblaze hasn’t noticed any reliability difference between the normal HDDs and the external ones, and (2) cloud services are designed with hardware failure in mind.

      Also, I don’t think everyone was loaded up with hard drives. HP and others noted the shortage might affect their bottom lines.

    2. Yev here w/ Backblaze -> Hi Mike! We actually use consumer level drives in our regular storage containers. Whether its good or bad, it works for us. This crisis did not force us to go from enterprise grade drives to consumer level drives, instead, we just found different ways of getting the consumer level products. Derrick’s points are great, and you can read about our storage container philosophy here: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/.

      1. How many redundant drives is a given piece of data stored on?

    3. Mike,

      I was an IT manager at a university during the shortage. I ran into the same difficulties even though I’d done proper capacity planning, and ended up stripping external drives for use in my private cloud for archiving. If my backup services failed, my users stood a good chance of losing critical data and years of research. (For those who wonder, yes, I was using CrashPlan.) I tested the hell out of them before making the decision and I can say for sure (with the exception of the WD Green drives) they were as reliable as the pro versions I was buying. There are many reasons to buy enterprise level HDDs; reliability is not necessarily one of them.

  9. I don’t get it; why are consumer hard drives cheaper? Especially considering they have extra hardware (the caddy and PSU), packaging, manuals, marketing, retailing store costs etc. that business customers do not need.

    1. I’m not in the hard drive industry, but if general business rules apply, it’s because regular consumers are a lot more price sensitive than business’. For example, a business knows they need the hard drives, if the price goes up, they still need them, and will have to get them at a higher price. Consumers are much more inclined to wait for a sale, or just not get the drive. The consumer prices on external drives increased too, but they went from about $110/$120 for a 3TB drive to $160/$170, instead of going from $120 for an internal drive to over $400.

    2. Enterprise drives are different. They maintain high throughput I/O rates when subjected to the high vibration environment of many servers and drives densely packed into racks.
      The performance hit is substantial.

      This was an interesting read…

      “Performance Impact of External Vibration on
      Consumer-grade and Enterprise-class Disk Drives”

      (Google the title)

  10. Great article that I found quite inspirational. Kudos to the folks at Backblaze for doing what they had to do to maintain their business growth. The picture of the hard drives at Vladik Rikhter’s front was absolutely priceless.

Comments have been disabled for this post