Blog Post

Facebook’s data center hardware fails less than others’ gear

[protected-iframe id=”15e56084c6dcd687c4edcf15e7b5980c-14960843-25766478″ info=”″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]

Session Name: Can Open-Source Hardware Reinvent the Data Center.

Chris Albrecht
Jo Maitland
Frank Frankovsky
Mark Tealey
Audience Member

Chris Albrecht 00:04

Thank you gentlemen. I think there are lots of things over the course of the two days that we’ll learn about that we can disagree on, things of different cloud approaches, different solutions, different ideas, different theories. But there’s one thing I think we can all agree on and that is our next guest has the best beard of anybody here. Jo Maitland and Research Director for GigaOM Research is going to be speaking Frank Frankovsky the VP of Infrastructure for Facebook. Its day two give me a break. For Facebook – I told him I would get his name right. I got his company wrong on pronunciation. They’re going to talk Can Open-Source Hardware Reinvent the Data Center. Please welcome Jo and Frank to the stage.


Jo Maitland 00:54

Hi everybody. So Frank, just fill in. How long has it taken to get to that length?

Frank Frankovsky 00:57

It was about Thanksgiving of last year that I stopped shaving. It’s been fun. Since I wasn’t blessed with the ability to grow it here, I do have the ability to grow it here, which I’ve just recently found out so it’s a lot of fun.

Jo Maitland 01:10

I just want to see a quick raise of hands from the audience as we get into this topic. How many of you are familiar with the Open Compute Project? So probably half.

Frank Frankovsky 01:23

More than I thought. That’s awesome.

Jo Maitland 01:23

Yeah. More than I thought. That’s really good. But for those people who aren’t familiar with it yet, Frank gives us a kind of the 10,000 foot view. What is Open Compute and also what do we mean by Open-Source hardware? We’re very familiar with Open-Source in the software world, but what is Open-Source hardware?

Frank Frankovsky 01:40

Open-Source hardware we essentially were really intrigued by what we saw happening in Open-Source software where people contribute to a common core and then they can get full transparency and extensibility out of that technology. The pace of innovation that was happening in software was just so much faster than what was happening in data center design or server design, and storage design. We decided let’s apply those same types of principles from software to the hardware space and see what happens. That was about two years ago and it’s been really interesting and exciting because there was so much pent-up demand for something like this. Because there’s so many people that care about their infrastructure and those same reasons that people so engaged Open-Source software to get that transparency and that extensibility so that can modify the software to their particular needs. That same flow is now happening down to the physical infrastructure, and that’s really where Open Compute comes in. It’s focused on the physical layer and open sourcing the best practices and the actual design and the source code, so to speak. Whether that’s the specifications the PCB layouts the CAD files those are all available out on so people can consume them, modify them and then contribute them back to the community.

Jo Maitland 03:11

So are we talking about right now Open-Source servers or Open-Source switches or Open-Source–which bits of the infrastructure are Open-Source and what’s still to come?

Frank Frankovsky 03:22

It, started with the work we did at Facebook where we designed our own data centers, or servers and those really– and the power distribution infrastructure was the first three contributions that were made by Facebook. Since then we’ve also Open-Sourced storage technology called Open-Vault and then most recently we also started an Open-Source project within OCP around network switches. We have these lovingly crafted islands of Open-Source hardware technology and then they were connected to the rest of the world through these proprietary black box switches. So we decided it was time to open up the appliance model of switches and design really a bare metal hardware switch, a pre-boot environment so that people can really make a choose – the best hardware and best software and bring those together at the switch layer. Unlike today where when you purchase a switch it comes pretty much preinstalled with that vendor’s software.

Jo Maitland 04:25

How much of Facebook is running on Open Compute today?

Frank Frankovsky 04:31

Our newest data center in Lulea, Sweden is actually 100% on Open Computer hardware, and that’s been a really awesome experience us. All of our data centers in the US have a combination of Open Compute as well as legacy equipment as well. One of the cool things that we’ve seen out of going a 100% to Open Compute is the quality level that we’re seeing in our Lulea data center. We looked at initial field quality rates of our US data centers versus our Sweden data center and failure rate–

Jo Maitland 05:07

Was that meantime between failure stuff, is that what you mean?

Frank Frankovsky 05:03

Exactly. Meantime between failure, or even a suspected failure. We actually measure tickets. So number of technician requests. In the US data centers where we have a mix OCP and non-OCP equipment failure rate averages around 3% and in Sweden it’s about 1%. So factor of 3X is a pretty big deal. I think it’s a pretty big deal whether you’re big company, because 3% of big number is a big number, but if you’re a small company it’s big deal because you have fewer servers and so you need the quality of those servers to be very high as well. So I think we’re proving out that Open-Source software now even Open-Source hardware can deliver very high quality experience.

Jo Maitland 05:51

Why is that? Is it just about standardization so if I for example built a brand new data center and completely standardized on hardware versus having a mishmash of different things. Could I achive the same effect or are you getting–what else are you getting with Open-Source hardware? Why are you getting that sense?

Frank Frankovsky 06:11

I think it’s two key things. I think one is removing all of the–we have this design tenant with Open Compute that it should vanity free device and what we mean by that anything that isn’t directly either calculating something–doing some amount of useful work. If it’s ancillary to the design it should be removed. Not only from a cost and an efficiency perspective because a lot of those ancillary features just consume power without doing useful work. But it also helps reduce the amount of material that you would decommission later on so it’s the best way to be sensitive to the environment is to actually eliminate materials on the front end. I think there’s a lot of great work going on in recycling and things like that, but eliminating materials on the front end is really beneficial way to put less waste into the waste stream later. I think eliminating a lot of those feature helps deliver higher quality and then the other thing is from a testing prospective because both Facebook and other adopters or Open Compute in the community test for their specific workloads. So the debt of testing that can be achieved when you’re doing a bespoke design is so much greater than–if you look at the challenge that some of the large incumbent providers have they try to design the smallest number of products to meet the widest possible market spec. That leads to pretty wide and swallow amount of testing. The ability to really do deep targeted testing from a QA prospective I think also leads to higher quality.

Jo Maitland 07:50

That’s interesting. I’m going to try not to geek out to much here, but I want to– if we were to open the box how different would it look too–I’m curious when you say you eliminate things that aren’t absolutely necessary this sort of vanity thing. If I opened up a generic server and an Open Compute server, what would stand out the most if I looked at them?

Frank Frankovsky 08:14

Probably the first thing you’d notice is that you don’t have to open the box. There’s no front bezel, there’s no lid on the server. We literally eliminated a lot of the plastic material. That’s one of the most–that was the easiest thing to do because those front bezels they’re really just there so somebody can put a brand on it. But they also impede airflow and so you actually have to spin your fans harder to get the air to pull through that silly plastic bezel – so we ditched that. We ditched top lid, which is just extra sheet metal material. So that would probably be the first thing you’d notice is there is literally there is no lid there is no front bezel. It’s like, “Wow.” It’s beautiful to me. I think a lot of people think, “Well if I brought my CEO through the data center that might not look so pretty.” But to me that’s the new beautiful. Then if you really want to geek out you can look the motherboard components themselves. We eliminate any of the proprietary management goop – technical term goop. We eliminate anything that leads our system-to-system administrators to any sort of incremental complexities associated with systems management. There’s a lot of features you can either populate or depopulate depending on the configuration; so it’s a more modular design. Then I’d say that one of the other things that’d you’d notice is different is that everything is serviceable from the front of the machine which is really important especially in free-cooled data centers. So one of the ways we achieved efficiency is we don’t use air conditioning in our data centers, which is great in you’re in the supply side aisle, it’s not so great when you’re in the exhaust aisle So we made everything serviceable from the front so our technicians don’t have to spend time behind the servers because it can get pretty hot back there. It’s defiantly a different kind of design pattern where everything’s accessible from the front. It makes it very easy to service at massive scale.

Jo Maitland 10:20

Faster presumably.

Frank Frankovsky 10:21

Much faster. We’ve done time and motion studies on how fast can I replace a DM in an OCP machine verses prior machines and it’s orders of magnitude faster. Because number one you don’t have to open a lid, but then getting to the equipment is just a lot faster and easier to service.

Jo Maitland 10:42

What impact has Open Compute had on this sort of legacy industry thus far? Are they sort of taking notice? What do you see, how does this percolated through the traditional suppliers?

Frank Frankovsky 10:57

It’s been really fulfilling to see the impact it’s having on some of the design patterns in the industry. The fact that Open-Source has been so successful in software and the fact that there’s more and more people that are passionate about really understanding about the physical layer of their infrastructure and taking hold of it and has caused entrenched suppliers to really take note and start to figure out how do we -as big technology suppliers – how do we embrace some of the things that are standardizing if not commoditizing at this physical layer and shift some of engineering resources to be able to–instead of inventing the next AC to DC power supply, because there’s only so many ways to convert AC to 12 volts DC. Or invent the next racking enclosure and start the–how many different flavors of chassis soup can you create? Move those engineering hours to something that’s more innovative instead of constantly reinventing the wheel at the physical layer. I think that that vision is there and I think that’s why we see suppliers embracing Open Compute and becoming members and starting to think differently and actually contributing IP to the foundation, which is just really cool.

Jo Maitland 12:18

We’re about five minutes guys so if anyone has questions that you want to ask Frank there’s a couple microphones kind of in the middle of the room. I don’t want to commandeer the whole thing myself, but I’m happy to if there’s no questions – nothing yet.

Frank Frankovsky 12:33

Looks like we have one.

Jo Maitland 12:34


Mark Tealey 12:36

Hi Frank. Mark Tealey. Hi Jo. Just curious relative to new data center failure rates versus the older data centers. First question on that was did you see the failure rates–were you comparing them specifically from your custom gear in the old data centers versus the custom gear in the new data centers or were you looking at all gear versus new gear?

Frank Frankovsky 13:00

It was both. We essentially we have five different server types. What we call type one, which is our web servers. Those were the same server type from US data centers to Lulea. Our type three, which is our data base server. We measured the delta between the database servers that we were using in the US – which were non-OCP to the database servers that we were using in Lulea and that’s really where the largest delta and quality was. There’s about a 5% failure rate on our machines in the US – the database machines and sub 1% in Lulea.

Mark Tealey 13:40

From a design perspective and I’ve looked at both and I apologize, I should have memorized this a little better. Do you see either a combination of design for the newest data center or a location therefore environment stability excreta having any impact on the availability of machines in either location?

Frank Frankovsky 13:59

We haven’t seen any delta in quality due to ambient environment kind of reasons. We have seen a small delta in efficiency. It appears that Lulea is running slightly more efficiently than – especially our East Coast, our North Carolina facility because it does get pretty hot and humid in North Carolina. It’s slightly very small fraction of a percent more efficient in Lulea just because it is so cool there. We actually use the waste heat from the servers to heat the office building most of the time in Lulea, which you can achieve a pretty low power utilization effectiveness in that kind of ambient environment.

Mark Tealey 14:44


Frank Frankovsky 14:45

Thank you.

Jo Maitland 14:49

Any other questions? Hi.

Audience Member 14:53

I was just hoping you would comment on the disaggregated server architect and your plans for deploying that and perhaps quantify the kind of benefits you expect to see and where you expect to see them.

Frank Frankovsky 15:06

Awesome. So you clearly keep up with the project. In January we started to outline this vision and some of the available technologies for what we call disaggregated rack, which essentially is a way to break up the currently monolithic design patterns that we have today in the industry where typically CPU, memory disk drive, NAN they tend to get bound to the same PCB – the same Printed Circuit Board – due to locality and latency issues. The challenge with that and as a hardware guy joining a software company like Facebook – the first thing that struck me was holy crap software moves so much faster than hardware. How do I shrink that and impedance mismatch between hardware and software? Because in hardware you have to pretty much make a guess because you’re working in the physical world – you have to make a guess months in advance of the software landing on the hardware so that you can start your supply chain and manufacturing logistics and everything. But then the software obviously changes on an hourly basis. The way users interact with the software changes on an hourly basis, but then we’re going to deploy this monolithic server – this bit of compute – that we really had to make a guess at. Sometimes it’s grossly underutilized, sometimes it’s well utilized. Sometimes we wish would have had more CPU or memory, or more disc because the software world and the way people interact with it are changing way faster than hardware. So the vision of disaggregated is rack is how can we create sleds of commodity components – CPU sleds, memory sleds, NAN sleds, disc sleds so that at the last hour as we learn more about the way the software’s going to exercise the hardware. We can modify the hardware much closer to the time of need. So it’s almost like a just in time kind of an inventory technique but applied at the technical level. And one of the core enabling technologies for that are really high speed low latency inner-connects between those components. So things that we announced at the January summit like silicon photonic shows a lot of promise because it has latency characteristics that theoretically that can allow you to disaggregate some of those things still – it’ll appear to the software as if there’s still bound to a single PCB. So we’re really envisioning kind of a next generation rack fabric that enables this disaggregated vision to occur. Raising utilization is one key part and I think the other key part of this is the ability to do more gradual or fined grained upgrades of the environment. So typically CPUs are on a path of every 12 to 18 months there’s a new CPU coming out. But in order to take advantage of that, if you’re on a monolithic system you typically wait and you depreciate the full system value before you go to the next CPU. If you can envision this idea of CPU and DRAM sleds being segregated, because DDR3 memory is good for multiple generations of CPUs – it’s technically feasible. You could do much less disruptive upgrades. There aren’t new promises. These kinds of things have been promised to us by the industry for many, many years. And there’s been some really beautiful but proprietary enclosed designs that have attempted this in the past. So the reason that I think this is going to be successful it’s the first time it’s ever been done in Open-Source.

Jo Maitland 18:36

So how far off? Where are we in the vision?

Frank Frankovsky 18:38

I would say that we should be able to demonstrate some pretty significant moves forward like proof of concept ready in the first half of next year.

Jo Maitland 18:51

That’s exciting.

Frank Frankovsky 18:53

It’s coming together nicely. The speed at which these things are happening is definitely picking up as well, as we issue these big challenges. The industry not just hearing it from Facebook, they’re hearing it from a whole community of people who have galvanized around this vision and say, “Hey, maybe Facebook is seeing this big storage issue before the rest of us. Or maybe they’re seeing it at a larger scaled than we are. But we all agree that this is a big problem and it’s currently under-served by the industry.” That really helps suppliers, because it makes those speculative investments a little less speculative. Because they’re now hearing it from a whole community of people that say if you build something like that we’ll actually buy it. So it helps to move things forward.

Jo Maitland 19:36

A question I think is how applicable is Facebook to the rest of the industry in terms of its requirements out of infrastructure. And it’s potential it’s, yeah. It seems to be more applicable, right?

Frank Frankovsky 19:53

It is increasingly more–if you believe that the trend toward cloud computing is going to continue then you would think that those design patterns would be applicable to a broader and broader set of people. Because whether they’re hosting at their own facility or buying it through the cloud that design pattern is going to continue to be more and more applicable to a broader audience whether it’s directly or indirectly through a cloud. Kind of the base level question for me is always well who wouldn’t want a more efficient server? So even if it’s not some of this disaggregated stuff and some of the things we just talked about, but it’s just, “Would you like your server to cost less or consume less power or put less waste into the waste stream?” The answers’ typically, yes no matter what size customer you are.

Jo Maitland 20:39

Frank thank you so much.

Frank Frankovsky 20:41

Thank you.

Jo Maitland 20:41

Always nice talking to you.

One Response to “Facebook’s data center hardware fails less than others’ gear”

  1. Greg Ferro

    I very interested in this discussion but wish I could have more information on the definition of failure. Is it hardware, software or could the overall solution have benefited from learning and experiences in previous data centres.

    Tantalising glimpses that I wish I could action.