Blog Post

RackSpace Outage Hits Home

[qi:014] Update: A truck driver because of a medical condition drove into a power transformer in San Antonio, Texas, this evening, causing it to explode. That explosion caused a major power disruption, and the power company in response cut power, which ultimately and took down RackSpace, our hosting company’s Dallas/Fort Worth based data center. Rackspace is based in San Antonio. This is the second time in less than a week that they have had power issues. Rackspace made the following announcement:

Without notifying us the utility providers cut power, and at that exact moment we were 15 minutes into cycling up the data center’s chillers. Our back up generators kicked in instantaneously, but the transfer to backup power triggered the chillers to stop cycling and then to begin cycling back up again—a process that would take on average 30 minutes. Those additional 30 minutes without chillers meant temperatures would rise to levels that could irreparably damage customers’ servers and devices. We made the decision to gradually pull servers offline before that would happen. And I know we made the right decision, even if it was a hard one to make.

Even though we are mostly hosted on, certain parts of the site are coming off of the RackSpace infrastructure. This prevented all our network sites from loading properly. Our email servers went down as well.

Everything seems to be back to normal, but it leaves me with one simple observation: our Internet infrastructure, despite all the talk, is as fragile as a fine porcelain cup on the roof of a car zipping across a pot-holed goat track. A single truck driver can take out sites like 37Signals in a snap.

39 Responses to “RackSpace Outage Hits Home”

  1. Marla Brady

    We have have several issues with rackspace including this outage. It was pretty much the last straw for us. We moved to Server Intellect and never looked back. I can and will always unstand outages that are BEYOND the control of the host, but when your paying actively for backup services only to find out when disaster strike you have no backups that is a problem.

  2. Seems that rackspace made a mistake and it was a costly one. If they have been a good company for so long though I think closing the book on them may not necessarily be the best solution. Mistakes happen I guess is all I am saying. Thanks for posting this article!

  3. Oh, I forgot to mention that although I have a 2 weeks retention managed backup, it seems that 2 weeks in rackspace time dimension is only 10 days as they cannot find any backup prior to the 14th, considering that yesterday we were the 23, I guess 14 days back would mean what ? 9th of January.
    So another mystery and another reason for me to worry and get few more grey hairs which suit me, but still …

    I will keep you posted about progress