12 Comments

Summary:

Facebook’s Timeline feature is the result of an engineering effort that was the equivalent of building a racing bike customized for a specific track, only without testing either until race day. At least that’s how it seems from a blog detailing how Facebook engineered Timeline.

Facebook-timeline

Facebook’s Timeline feature is beautiful, although some revile it. But love it or hate it, Timeline is the engineering equivalent of building a racing bike customized for a specific track, only without testing either until race day. At least that’s how Ryan Mack, an infrastructure engineer with Facebook, makes it seem on a blog posted Thursday detailing how Facebook engineered its Timeline feature.

Making a legacy MySQL database faster.

The blog post also offers a ton of data on how Facebook dealt with petabytes of user data stored in legacy MySQL systems on slow disks that could make Timeline less responsive. The company implemented a new architecture that separates out older data to slower disk storage and keeps more recent and more accessed data stored in flash drives and cached using memcached. From the blog:

Before we began Timeline, our existing data was highly normalized, which required many round trips to the databases. Because of this, we relied on caching to keep everything fast. When data wasn’t found in cache, it was unlikely to be clustered together on disk, which led to lots of potentially slow, random disk IO. …A massive denormalization process was required to ensure all the data necessary for ranking was available in a small number of IO-efficient database requests.

Here’s a visual of the system:

Mack spent a lot of time detailing the challenges of denormalizing the data, which entailed getting an intern to define a custom language that would tell a compiler how to convert old data into the new format and the use of three “data archeologists” to write the conversion rules. It also required Facebook to move older activity data to slower network storage while maintaining acceptable performance. To do this Facebook, “hacked a read-only build of MySQL and deployed hundreds of servers to exert maximum IO pressure and copy this data out in weeks instead of months.”

To speed IO even further, engineers consolidated join tables into a tier of flash-only databases. Since PHP can perform database queries on only one server at a time, Mack said Facebook wrote a parallelizing query proxy that allowed it to query the entire join tier in parallel. Finally Facebook attempted to future-proof its data model, but we’ll see how far that takes it.

In response to a question I sent Mack, said through a spokesman, that it took Facebook two weeks to dump everyone’s old profile data from the existing system into an archive using “some off-the-shelf network attached storage.” That data was then stored on “the new servers that now power Timeline, which, at the time, were fresh and didn’t have anything hitting them from production. We did incremental updates weekly until later, when we had writes going to both systems in real time.” That’s a lot of fresh servers, although Facebook remained mum about the exact data transferred.

Facebook also built the Timeline aggregator to run locally on each database to avoid information traveling over the network unless that information will be displayed on the page. That’s another timesaver. It also used it’s Hip Hop code for speeding up PHP in that aggregator.

Building in parallel.

The other element of the blog that I found amazing was how Facebook apparently denormalized its data, had a product team visualizing the Timeline UI, and built a scalable back end system for everything to run on all at the same time. I’m going to call that crazy, but Mack notes that’s what kept the development time frame to a mere six months.

I asked Facebook about what it learned during that process, because I imagine a lot of other sites would love to be able to develop their front and back end systems in parallel and save the cost of replicating their entire production environment’s data to test it. The response a spokesman emailed me from Mack was as follows:

Layering is a good way of describing the process of building Timeline. To make sure we could work in parallel, we always made sure that there was at least a temporary technology that each part of the team could build on top of. Some of these were pre-existing, but many of them were quick prototypes built in a couple of days as needed. We also had frequent scrums to identify places where people were blocked on each other, and everyone on the team did an excellent job of knowing what others needed or would soon need.

When the production solutions were ready, there was always some friction in migrating from the temporary solution, but usually it was just a few days of work. It was a definite win compared with the weeks or months of delay we’d have faced if we had waited for the production solution up front. Given that we wanted to build the entire system in six months, it was critical that we avoided wasting any time.

From nothing to Timeline in six months is pretty impressive, considering we’re talking about creating something to support more than 800 million users. Now you know a bit more how this Timeline sausage was made.

  1. Interesting story, thanks. BTW For the Timeline haters who want the old style profiles back, there is a way to get rid of timeline( though its most likely only temporary until facebook fix it ) For more info go here: http://fbpurity.com/disable-facebook-timeline.htm

    Share
  2. Hi Stacey, timeline is an interesting engineering challenge indeed. Any idea on how many engineers worked on this? Or the amount of work effort behind this? Thanks.

    Share
  3. Mehdi Naseeruddin Thursday, January 5, 2012

    A huge challenge as I see it, especially reenginering the complete datawarehouse. It will be exciting just how many people, servers, databases, tables we’re talking about. This could make a great infographic. Cheers to the FB team.

    Share
  4. Michael James Lester Thursday, January 5, 2012

    It has a button for editing a post, but the only thing that you can edit is location and date. How about allowing users to edit the spelling of what they typed? Didn’t anyone think of that beforehand?

    Share
    1. That ‘ll defeat the purpose :)

      Share
    2. Editing the content of your status updates etc isn’t possible anyway, so there’s no reason for Timeline to change that. Consider the case of someone posting an innocuous statement that several friends like. Then change it to a homophobic slur. A common friend (or perhaps a stranger if it was a public post or a friends of friends post) sees the activity in Ticker and clicks on it – they see that their friend liked a homophobic slur, not the innocuous statement.

      Now, there are ways around that – only allow a % of the letters to change, or display other visual indicators that the post has changed and provide a way to access the original. However, I’d guess that’s only the tip of the iceberg, and there are more considerations that the engineers are aware of that I’m not aware of.

      Share
  5. From a technology standpoint this is just a super cool feature. Facebook has done it again.

    Share
  6. Welcome, join the conversation

    Share
    1. Let’s be honest, the timeline was a great success (technically).
      it must have taken a lot of coding and planning.

      However, it does not solve the user’s visual needs at all. It actually makes the site more confusing and distorts the navigation system.

      these are the blunders that open room for competitors, Which is why i’m thinking of creating my own social network using facebook’s old style.(since they dont want it anymore) :)

      Share
  7. Roman Tarnavski Tuesday, January 10, 2012

    Great effort, and result. Would be superb for other development teams to be exposed to lessons learnt, I hope Facebook hits the ‘share’ button on this internal process.

    Share
  8. Jennie Dillingham Wednesday, January 25, 2012

    how do i start my facebook timelime>it dont pop up on my facebook no more>can you help me?thanx jen

    Share
  9. Someone is making money off of this issue.
    http://danyarstudios.com/?p=347

    Share

Comments have been disabled for this post