9 Comments

Summary:

Twitter will move into its own data center soon as it seeks to scale its social messaging service. Speaking at the Chirp developer conference yesterday in a session on scale, John Adams, a Twitter engineer, laid out Twitter’s strategy to keep the fail whale at bay

Twitter will move into its own data center soon as it seeks to scale its social messaging service, according to a presentation by John Adams, one of the messaging service’s operations engineers. Speaking at the Chirp developer conference yesterday in a session on scale, he laid out Twitter’s strategy to keep the fail whale at bay — which included plans to soon move to its own data center (see slide below).

It’s unlikely Twitter is building and operating this data center itself. In 2008 Twitter signed a contract to host its servers at NTT because the latency in the cloud was too high for its service, and last August NTT said it had leased 15,000 square feet in Santa Clara, Calif. for expanding its data center operations in part because of Twitter’s success. So my hunch is that Twitter is moving into NTT’s data center dedicated to the messaging service, as opposed to building and operating its own, which would take a while. I’ve reached out to NTT and Twitter for more information. In 2008 Twitter saw year-over-year traffic growth of 752 percent and from 2008 to  2009, traffic rose 1,358 percent. It serves 55 million tweets a day.

Twitter has gone from hosting aspects of its service on Amazon Web Services and via Joyent to discarding the cloud because latency was too high (we’re going to have a talk about improving latency in the cloud at our Structure 10 conference in June). Essentially, like most IT folks, Twitter’s operations group plays a game of whack-a-mole attempting to add hardware, improve algorithms or add new code to solve the next engineering bottleneck.

Even if the dedicated NTT data center space manages to improve the service for a while, will Twitter ever end up following Facebook, which this year said it would build its own data center so it could control costs and reduce its energy consumption? And what does this say about the evolution of scalable infrastructure?

Chirp 2010: Scaling Twitterhttp://static.slidesharecdn.com/swf/ssplayer2.swf?doc=johnadams-scalingtwitter-100416061113-phpapp01&rel=0&stripped_title=billions-of-hits-scaling-twitter
View more presentations from John Adams.

You’re subscribed! If you like, you can update your settings

  1. What’s Up With Twitter’s New Data Center? – Gigaom.com | Twitter Traffic Experts Friday, April 16, 2010

    [...] Post By Google News Click Here For The Entire Article Twitter Traffic Machine Review- April 16th, 2010 | Category: Twitter Traffic Machine [...]

  2. Stacy — You’ve misunderstood. Twitter is not building their own datacenter physically like Facebook, Google and MSFT do. They are just moving out from under NTT/Verio. Likely to one or more carrier neutral facilities like those provided by Equinix, Switch and Data, TelX, TeleCity, etc.

    From what I’ve gathered, they are currently using a managed hardware service solution from NTT which was great early on, but which has hurt them as they’ve tried to scale and “get closer to the metal.” It’s also much more expensive than doing it yourself, even when renting the physical space and power from others.

    Their arrangement with NTT limits their ability to run their own network, under their own ASN, with their own IP space. They can’t route, peer, or do other fancy network tricks. It’s probably time they start doing that if they want to continue their march towards operational excellence.

    It’s quite possible they’ll keep NTT managed services around for a while, and even buy transit from NTT America, but it’s time for them to run their stuff mostly on their own gear.

    1. Stacey*, of course. Sorry.

      (can’t edit my own comment)

  3. I don’t think it’s fair to say that Twitter are not using Cloud Computing because of latency issues – that’s just something they experienced with the Cloud providers they used and the particular infrastructure setup those providers have in place.

    Its quite possible to have excellent latency and IO (the OTHER Cloud issue) in the right setup – it just might cost you a bit more – but you would still benefit from the utility and scalable nature of Cloud Computing.

  4. Twitter Getting Own Data Center? Friday, April 16, 2010

    [...] center (as per operations engineer John Adams’ presentation at Chirp this week). As Gigaom points out, one possibility is that Twitter will move into NTT’s data [...]

  5. Jimmy Sou twitter and marketing Friday, April 16, 2010

    I think Twitter will build its own datacenter especially now they are moving to advertising and they seem to want to be in more control

  6. TWiCC #10 with Gavin Doughtie & Jason Baptiste Monday, April 26, 2010

    [...] Story 5 Twitter recently announced at the Chirp conference that it’s getting its own data center. [...]

  7. Twitter to Build Data Center to Beach the Fail Whale Thursday, July 22, 2010

    [...] wondered if Twitter was going to build its own data center back in April after John Adams, a Twitter engineer, mentioned the proposition on a slide. However, I ultimately [...]

  8. They are building there own Data Center. Facility Gateway Corp is the builder. No more managed hosting for twitter

Comments have been disabled for this post