Comments on: RackSpace Outage Hits Home

By: Royal Pingdom » Truck takes down Rackspace

Royal Pingdom » Truck takes down Rackspace — Wed, 11 Jun 2008 15:56:47 +0000

[...] GigaOM, Data Center Knowledge, Valleywag, [...]

By: Marla Brady

Marla Brady — Mon, 09 Jun 2008 17:17:05 +0000

We have have several issues with rackspace including this outage. It was pretty much the last straw for us. We moved to Server Intellect and never looked back. I can and will always unstand outages that are BEYOND the control of the host, but when your paying actively for backup services only to find out when disaster strike you have no backups that is a problem.

By: Truck Supplier

Truck Supplier — Tue, 27 May 2008 19:30:04 +0000

Seems that rackspace made a mistake and it was a costly one. If they have been a good company for so long though I think closing the book on them may not necessarily be the best solution. Mistakes happen I guess is all I am saying. Thanks for posting this article!

By: Pascal Brenner

Pascal Brenner — Fri, 25 Jan 2008 05:00:35 +0000

Oh, I forgot to mention that although I have a 2 weeks retention managed backup, it seems that 2 weeks in rackspace time dimension is only 10 days as they cannot find any backup prior to the 14th, considering that yesterday we were the 23, I guess 14 days back would mean what ? 9th of January. So another mystery and another reason for me to worry and get few more grey hairs which suit me, but still ...

I will keep you posted about progress

Pascal

By: Pascal Brenner

Pascal Brenner — Thu, 24 Jan 2008 17:09:14 +0000

I was a big fan of Rackspace always telling anyone who were happy to listen, even to those not willing to ;-))

But November was a first loss of 150% trust, then this week this is the end of it. We had our server potentially compromised, we then decided with the advice of Rackspace engineer to rebuild the server.

Now it took far too long over a day, then we realise that we didn't have backup after the 16th of January, so 5 days with no backup, and yes we do have Managed Backup with Rackspace.

So now I am with a server which is partially restored, emails are back online but we have lost 7 days of them which is significant. And on top of that we have lost one very precious directory whereas the data was a reference and no other backup or copy because it was confidential and was supposively backedup.

My question to rackspace is how come a Managed Backup remain un-noticed for 5 days. I have told them this of course. How come there is no alert defined if the volume backed-up is suddenly less than 50% of the normal volume ?

All this to say that I am looking actively at the moment in finding another host for a dedicated server as my level of trust reach the bottom.

I have actively defended rackspace at our board of director, but this time, I can't see what excuse I can find for this.

Sorry guys at Rackspace, but not good enough ;-(

Pascal

PS : I am not working for any competitor of Rackspace and I do not have any friend / family / acquaintance with any competitors or related companies to a competitor. I say that in case someone think that it might be the case.

By: SilverDock blog: Product Video Merchandising, Video SEO, and Video SEM for Online Retail and E-Commerce Marketing » The Internet is a giant game of Jenga

Fri, 18 Jan 2008 20:32:26 +0000

[...] fall they suffered two outages due to power issues — the most colorful one was caused when a medically-incapacitated truck driver drove into a power transformer outside their data center. Today, 37signals suffered a two-hour Rackspace hardware outage in the [...]

By: Xbox Live maintenance tonight » XBox Game Players » Xbox Live maintenance tonight

Sat, 17 Nov 2007 13:35:20 +0000

[...] looks like this is what happened with our host last night. They did fail over to the generator eventually, but they couldn’t do it instantly [...]

By: Peter

Peter — Wed, 14 Nov 2007 21:23:21 +0000

I need to add that placing all your trust in one place is dangerous. No one can predict or defend against every scenario. Rackspace is head and shoulders above any other hosting company I have used but that should never replace thorough and detailed disaster planning and testing. I am using this situation as an incentive to do just that.

By: Peter

Peter — Wed, 14 Nov 2007 21:09:55 +0000

We have hosted with Rackspace for 3 years and they have been fantastic. This outage really hit us hard though. It nuked the boot drive on a RAID array of our database server and our 200 customers were offline for 21 hours. We worked all night and all day to restore the DB environment. It has definitely shaken the trust of some of our customers. This could put a fragile company out of business.

By: Graham Weston, Chairman of Rackspace

Graham Weston, Chairman of Rackspace — Wed, 14 Nov 2007 20:26:45 +0000

Lew hit it right on the head Om. We let you and many of our customers down, and for that I am sorry. We will continually update our website here http://www.rackspace.com/information/announcements/datacenter.php so that customers can know what we are doing to fix the problem.

By: A Voice of Reason

A Voice of Reason — Wed, 14 Nov 2007 15:17:01 +0000

Dear Matt/Jim/Bruno,

I'm truly impressed in the trust of your neighbor's recommendation and your ability to negotiate a contract and migrate your services over to Verio in less than 24 hours.

Next time you try to slam a competitor be sure you don't leave your name and company URL in your signature file. Your post is full of lies (and spelling errors to boot).

Nice try.

By: John

John — Wed, 14 Nov 2007 09:42:26 +0000

Did anybody actually read Rackspace’s comments on what happend? The truck did not cause the outage. It was the power company.

http://www.rackspace.com/information/announcements/datacenter.php

6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended. When we transferred power to our secondary utility power system, the data center’s chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim.

By: Neto

Neto — Wed, 14 Nov 2007 07:27:51 +0000

This happened in Dallas not San Antonio!!

By: Mark Imbriaco

Mark Imbriaco — Wed, 14 Nov 2007 04:35:07 +0000

Matt: When I spoke to my account manager at Rackspace, I asked that very question. From what I understand, the driver is doing fine.

By: JohnM

JohnM — Wed, 14 Nov 2007 03:38:14 +0000

Suddenly Leigh Anne wants his 99.999% advertisement removed :)
Anybody who has worked in a datacenter knows that there is only so much you can get redundant without the cost rising.

By: Web-Tones

Web-Tones — Wed, 14 Nov 2007 02:44:25 +0000

Uptime: Serious Business Indeed!

Link: RackSpace Outage Hits Home � GigaOM. No need to pile on here, but clearly this is an issue that is going to become and more and more important going forward. It is not clear whether this could have happened