If Facebook VP of Technical Operations Jonathan Heiliger were a superhero, his name would be “Scaler-man.” Today the service has something like 500 million users, twice as many as last July. But even so, Heiliger admits that the company should have planned its infrastructure better for the challenges it faces now.
The biggest lesson he’s learned in the past year is “you can never think about scale too early,” Heiliger said at GigaOM’s Structure conference today. “We’ve thought really far ahead but we’ve also punted on really critical things that we needed to do,” he said. “Now we’re under the gun rather than being able to do them on our own time.”
Heiliger didn’t specify what Facebook had punted on, however he mentioned the company has replaced some of its larger systems such as the way it serves platform traffic. Facebook has also undertaken recent projects to improve site speed. A couple months ago the company killed its “Facebook Lite” product (a re-engineered version of the site built to be extremely fast), with the explanation being partly that the learnings from building a streamlined version had been incorporated into the main product.
But scaling Facebook is also about continual evolution, said Heiliger. A huge part of the service’s infrastructure success comes by way of benefiting from and contributing back to open-source technologies, he said, naming HipHop for optimizing PHP code and Haystack for managing its massive amounts of photos (1 million delivered per second, expected to be 10 times that within a year, he said).
Facebook primarily relies on its own infrastructure because it likes the agility, performance and cost reductions that come from control, Heiliger said. (On the negative side, that’s led to the unforeseen “bottlenecks” that Heiliger said he regrets punting on.) The company is now building its first data center in Oregon, which Heiliger said is due to be turned on soon. But he advised young web startups to do the opposite — at least to start.
“For a consumer web site starting today I would absolutely use the cloud,” he said. “I wouldn’t hire someone like me — we’re really irritating, cynical people and we make silly remarks at conferences.”
Heiliger was referring to widely publicized comments he made a year ago at Structure 09, where he threw down the gauntlet for semiconductor and hardware makers, saying they weren’t up to the task of serving the needs of enormous, growing web businesses like Facebook.
Today, Heiliger said those companies have come a long way in the last year. Facebook recently released a benchmark paper on CPUs, finding a 45 percent improvement in dollars per watt between this generation and the last. Heiliger also mentioned startups such as SeaMicro and Tilera and their promising multicore efforts, saying Intel and AMD were also pushing similar projects. (The CTOs of both SeaMicro and Tilera spoke at Structure yesterday.)
However. Heiliger invited cloud hosting providers to contact him with proposals. “At Facebook we look once or twice a year if we should put more infrastructure in the cloud or ourselves,” he said.
Please see the disclosure about Facebook in my bio.