Blog Post

Two days after Amazon’s notice went out, Rackspace warns its customers of Xen-focused reboot

At 9:20 p.m. Friday night, PST, Rackspace sent out an email to its cloud customers alerting them of planned cloud server resets to start this weekend. This was two days after Amazon Web Services notified its customers. The reason for the do-over was soon determined to be an unspecified Xen issue, a fact that Amazon acknowledged on Thursday, a day after it alerted its users. More here on the Rackspace community page, which posted at  1:01 am. September 27.

Per the [company]Rackspace[/company] email, all standard, performance 1 and performance 2 cloud servers will be affected and Rackspace recommended that customers proactively:

  • Verify all necessary services (Apache, IIS, MySQL, etc.) are configured to start on server boot
  • Ensure that you have up-to-date server images and file-level backups enabled, and confirm that you have backups of all critical data
  • Confirm that any unsaved changes, such as firewall rules and application configurations, are indeed saved

There is more on the Rackspace status page. To recap, on Thursday evening, Rackspace had no statement or response to questions about the Xen issue although it was pretty clear its public cloud — which relies on Xen hypervisor — would be affected.

For those who did not get the Rackspace email here’s a screen grab showing the tentative schedule:

Cloud pundit Ben Kepes, in a Forbes post, called this an example of how not to respond to a critical fault.

 

Feature photo courtesy of Flickr user Steve Snodgrass

2 Responses to “Two days after Amazon’s notice went out, Rackspace warns its customers of Xen-focused reboot”

  1. Xen have a pre-disclosure list[1] for large deployments of Xen to get critical notifications before they become public. This is very much like the Heartbleed bug where Google and a few others got pre-warning and were able to have their patch in place before it was publicly announced. Amazon are on this list, and they announced these maintenance reboots several days before Rackspace.

    But Rackspace are also on the list! This means they had the same notice Amazon did[2] yet waited until late Friday to alert customers of the reboots that same weekend. What happened here? You could be cynical and say they deliberately withheld the details to the end of the news cycle late Friday, but more likely they didn’t have a proper process in place to handle critical vulnerabilities.

    [1] http://www.xenproject.org/security-policy.html
    [2] https://twitter.com/blueboxjesse/status/515734010207563776

  2. Eamon Walsh

    Unless you’re a gaming company planning a new release and a public cloud burst, yur e-commerce might not sink, but you’re still likely to feel the outage bigtime. Here’s hoping cloud emerges stronger after this reboot.(ibm.co/1qOBpOW)