“Due to an error in a routine change, Twitter was not available from 1:08pm PDT to 1:33pm PDT.  We rolled back the erroneous change as soon as we identified the issue. Additionally, some users may have experienced Tweet delivery delay from 1:33pm PDT and 1:53pm PDT.  We apologize for the inconvenience.”

A nearly 30 minute update is a substantial amount of time, particularly considering that the outage occured around the time the markets closed. Anecdotally, we heard that some users were able to get tweets through the Chrome app for Tweetdeck during this time, but people’s experiences were mixed.

It’s been a while since we’ve had a lengthy Twitter outage, as the days of constant fail whales seemed to be over. We wrote extensively in 2011 how Twitter’s shift to its own custom-built data center made significant headway in cutting down on those types of delays.

But nearly 30 minutes of delays in 2013 is a different story than it might have been just a few years ago. Now that tweets are flowing through Bloomberg terminals and can affect stock prices in seemingly an instant, the stakes are much higher for Twitter if the site goes down. Not to mention the advertisers and media companies Twitter is now accountable to.

Note: This story was updated and revised substantially to reflect changes in the information from Twitter and the status of the outage.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• NewNet Q2: Google closes the quarter with a bang

A spokeswoman confirmed the outage, but had few details:

“We are currently experiencing a disruption with our Internet service. We are working with our vendors to restore service as soon as possible and apologize for any inconvenience this has caused our customers. We will continue to work towards resolution until all issues are resolved.”

Earlier this year in January, AT&T experienced a substantial service outage affecting thousands of its customers, and lasting for a few days. The culprit in that outage was a software update that affected some, but not all of AT&T’s 7.4 million U-verse subscribers.

While power outages, fiber cuts and even software updates can take out a subscriber’s broadband for a bit, nationwide outages are rare. Maybe tomorrow on its earnings call, CenturyLink will explain what happened to cause this one. Meanwhile, a mobile hotspot or event a project like the recently launched BRCK, would come in handy right now for CenturyLink’s subscribers.

Updated 12:20pm: A CenturyLink representative said all service has been restored as of around midday Pacific Time.

Updated at 1:20 pm: CenturyLink said it has identified the cause of the outage as a problem with its core routers, but is investigating the problem further.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

According to a Wednesday Outlook blog post by Microsoft VP Arthur de Haan:

“On the afternoon of the 12^th, in one physical region of one of our datacenters, we performed our regular process of updating the firmware on a core part of our physical plant. This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way. This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.”

Many Hotmail and Outlook users have reported on-going issues with the services since January, when Microsoft started migrating Hotmail users over to Outlook.com. Making things harder to track is that not all these issues show up on the Microsoft Live Status page which only reflects problems affecting a “significant” number of users. Microsoft told users who are having issues to log into their account to see for more information on their status.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang
• From car to cloud: the future of the in-vehicle app landscape

There was another public flare up of problems at the end of February. At that time, a Microsoft spokeswoman explained that when a small number of users are affected the status page will not show a problem. Issues that impact a significant number of customers, on the other hand, “will be noted and visible on the server status page.”

#hotmail and #outlook have been down since 2:35pm PDT. See status.live.com/detail/outlook.—
  (@edanderson) March 13, 2013

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

@ronan_kelly We are aware the Hotmail/Outlook services are down and doing our best to fix it. See spr.ly/6010nTnk for updates. ^KLM—
Microsoft Support (@MicrosoftHelps) February 25, 2013

This is just more bad news for the popular Hotmail email service that suffered a big snafu on January 8, and folks don’t have much of a sense of humor about it.  Indeed some GigaOM commenters say the problems have persisted for them since that time — all the while the Microsoft status page registering no issue.  The UK’s IT Pro reported on the latest issue here, laying the blame on the snafu on the ongoing migration of Hotmail users over to Outlook.com.

An outage is definitely not good but  what’s really troubling here, as ITPro points out, is that the Hotmail/Outlook.com status pages reflect no problem at all. If cloud service purveyors want customers to come aboard, they need to provide a real window into what’s going on — the good and the bad. And this lack of disclosure — except thank goodness for Twitter — is unacceptable.

Grrrrrr come on Outlook it's been 3 hours now sort it out. #outlook #hotmail—
Emma Akbareian (@emmaakb) February 25, 2013

I will update this story when Microsoft responds to request for comment.

Update: On May 2, 2013, Microsoft said the Hotmail-to-Outlook.com migration is complete. Please feel free to comment there.

Update: According to Microsoft:

“The team is currently investigating an issue affecting a small number of users’ access to Hotmail and Outlook.com, and they are working to restore full access to the service as quickly as possible. I will connect with the team on the status page, as I know full transparency is important to the team.”

Another update from Microsoft spokeswoman re. the unchanging status page:

“If there is an issue impacting a small number of customers, only affected customers will see the personalized status update on http://status.live.com. Customers must be logged into their account to see this status update. The issue today is only impacting a small number of users, so only affected users will see an update on the status page. However, issues that impact a significant number of customers will be noted and visible on the server status page to everyone, regardless of whether you’re logged into your account or not. Microsoft takes both issues and full transparency related to issue updates very seriously and works to resolve issues quickly for customers.”

This story was updated at 10:11 a.m. PDT with Microsoft comment; the second update pertaining to the Hotmail status page was madea t 12:13 p.m. PDT.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Public, private or hybrid? How to move to the cloud
• A near-term outlook for big data
• Flash analysis: the future of Yahoo

But that doesn’t mean you can stop worrying about hardware redundancy or high availability data centers. As cloud applications get more complex, you must add complexity at the hardware and infrastructure level. The key is that complexity will be under your control, which will help your application stay resilient.

Resiliency isn’t just a buzzword anymore

The very complexity of modern distributed or even vertically integrated solutions (I.e., Amazon vs. VCE vs. legacy IT architectures) means that you can’t accommodate all risk vectors. Adrian Cockcroft of Netflix is a brilliant cloud use/application design strategist, and he has many other incredibly skilled people working with him, yet Netflix was still unable to account for every variable that might occur in the complex system (Amazon) that supports his complex system, causing an outage affecting millions of customers on Christmas eve.

I’ve been designing, building, implementing, and operating critical infrastructure for enterprises for over twenty years now. In that twenty years I’ve uncovered a large number of truths about the IT world, and one of those truths is “you can’t assume anything”. Every time a server is installed, a switch configured or a new update applied to an application a change has occurred. The change might have been tested under every known scenario (impossible), yet still fail because of a heretofore unimagined use case. In my experience, it’s often the combination of activities that occur at the worst possible moment that cause you problems.

Go hybrid, young developer. Go, hybrid.

“ <a title="Attribution-NoDerivs License" Photo courtesy of Flickr user piermario

There has been considerable discussion about resiliency lately, and in some cases the underlying concerns around management and performance of complex systems has been well documented. One of the best discussions on the topic of resiliency was by Richard Cook.

Mr. Cook points out, as I’ve written in the past, that your operations practices and people are often the keys to your success. What I would also argue is that when you consider your design, locations, people and process you should include strategy for system reliability beyond mere application resilience.

Many of the companies I work with have global organizations with tens of thousands of internal and or external customers. In all cases these companies have decided that one cloud isn’t the right approach. A hybrid approach, while more complex in the beginning, offers significant upside in several areas over the long term besides higher reliability, including avoiding lock in, and having the ability to deliver service to any region, regardless of the available cloud provider. These same companies have also decided that carrying the risk that their most important IT assets are down through acts of god is also unacceptable.

Any large system has a risk that a single problem can cause a cascading effect on its functionality or availability, as has been the case with Amazon and Google among others. One of the best ways to avoid this cascade effect is to ensure that some of your applications or portions of each application run on different platforms. Having different platforms helps to ensure that problems propagating in any one cloud platform won’t affect applications in the other.

What apps need a hybrid cloud?

As I’ve said before there is no one size fits all cloud or IT strategy. Each application or service has to be evaluated against traditional risks of outage vs. loss of business. The vast majority (70 percent or more) of the applications in an enterprise IT org don’t require anything approaching 100 percent uptime. Of the 30 percent that are critical, many will likely support the business just fine with a 99.95 percent uptime design. What you have left is the 5-15 percent of applications that the business really considers “mission critical”.

It’s also true that in some cases just 5 percent of applications could account for 50 percent or more of your company’s IT work output. So the number of applications is less important that their scope and importance. This differentiation of applications is critical to help IT teams make the decision about what applications need the resiliency of more than one cloud platform.

High reliability doesn’t have to cost double the infrastructure and people costs like you might expect from historical efforts in legacy IT. The ability to provide redundant environments or splitting your applications between two environments is a real option today when using cloud. That’s not to say there won’t be extra costs, but the costs are more related to startup investment than they are to on-going hardware and environment management and replacement costs.

In other words, if your environment is big enough or critical enough spending some upfront effort and cash to enable a hybrid cloud environment (I.e., private & public cloud or two different public environments) can’t easily be justified. In fact, there’s an online gaming company that is using the “any cloud” approach as part of their business model. The use of multiple clouds allows it to grow quickly where and when it needs to while supporting latency and availability requirements.

Complexity you control versus complexity you can’t

Once you’ve taken the steps necessary for adding another cloud provider and made multi-cloud work for critical app, it’s now much less of a hurdle to make them work for less critical environments. I also want to make clear that you can distribute one application across several clouds or split multiple applications across several clouds. The result of the multiple applications on several different clouds is lower reliability for any one application, but higher reliability for your applications as a whole. In other words, you’ll have avoided the risk of having all your apps down at the same time.

I know it seems counterintuitive to create more complexity to help protect you from complexity, but it’s not. The complexity you’re creating can be easily managed with the appropriate up-front effort and it’s under your control. On the other hand, the complexity you accept for your critical app running on one cloud is completely out of your control.

So go forth and build sustainable, supportable, highly reliable application environments across providers, in the long run it’s the smart and safer thing to do.

Mark Thiele is executive VP of Data Center Tech at Switch, the operator of the SuperNAP data center in Las Vegas. Thiele blogs at SwitchScribe and at Data Center Pulse, where is also president and founder. He can be found on Twitter at @mthiele10.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Migrating media applications to the private cloud: best practices for businesses
• Cloud computing infrastructure: 2012 and beyond

All told, the outage (which began at 12:24 p.m. PT) lasted 23 hours and 41 minutes and, at its peak, crippled 6.8 percent of load balancers in the region while leaving others running — albeit unable to scale or be modified by users. The Elastic Load Balancer team didn’t realize the root cause of the problem for several hours, at which point it began the challenging process of attempting to restore the state data to a point in time just before its accidental deletion. At 12:05 p.m. PT on Dec. 25, AWS announced that all affected load balancers had been restored to working order.

AWS says it has taken multiple steps to ensure this situation doesn’t repeat itself, or at least can be resolved faster should something similar occur. The first — and likely easiest — fix was to incorporate stricter access control to production data of the type that had been deleted. According to the AWS report, that’s typically the case, but the company “had authorized additional [Elastic Load Balancer] access for a small number of developers to allow them to execute operational processes that are currently being automated.”

On the technological side, the company had the following to say:

We have also modified our data recovery process to reflect the learning we went through in this event. We are confident that we could recover ELB state data in a similar event significantly faster (if necessary) for any future operational event. We will also incorporate our learning from this event into our service architecture. We believe that we can reprogram our ELB control plane workflows to more thoughtfully reconcile the central service data with the current load balancer state. This would allow the service to recover automatically from logical data loss or corruption without needing manual data restoration.

More shocking than the AWS outage, though — they’ve happened before and will almost certainly happen again — is that the Christmas Eve outage actually took down Netflix, which is often cited as the most-advanced AWS user around. It has a host of homegrown tools built specifically for the purpose of monitoring, managing and adding reliability to its AWS-based infrastructure. There’s a reason even President Obama’s tech team relied on the company’s best practices in order to keep its campaign applications up and running during election crunchtime.

Adrian Cockroft (center) at Structure 2012
(c)2012 Pinar Ozger pinar@pinarozger.com

In a blog post on Monday, Netflix cloud guru Adrian Cockroft acknowledged the effects on company’s streaming service and explained how it affected different devices in different ways. Cockroft also provided a mea culpa of sorts, explaining that while Netflix has an impressive track record when AWS outages are confined within Availability Zones, challenges still remain when outages affect significant portions of AWS regions as this one did.

Indeed: Netflix recently survived an outage in October, but was hit by a July outage in which a cascading bug spread across Availability Zones in the US-East region. “We are working on ways of extending our resiliency to handle partial or complete regional outages,” Cockroft wrote.

However, he cautioned, figuring out how to do it correctly will take some work given the complexity of cloud computing infrastructure:

We have plans to work on this in 2013. It is an interesting and hard problem to solve, since there is a lot more data that will need to be replicated over a wide area and the systems involved in switching traffic between regions must be extremely reliable and capable of avoiding cascading overload failures. Naive approaches could have the downside of being more expensive, more complex and cause new problems that might make the service less reliable.

At this point, though, if anyone can figure out build reliable cross-region services on Amazon’s cloud platform, it’s probably Netflix. Although, AWS and other cloud providers will certainly undertake their own work to improve reliability across global data centers, thus making themselves all the more appealing to potential customers.

We’ll have to wait to see how the lastest in a string of 2012 outages for AWS affects CIO sentiment toward the cloud or if, like Cockroft, they’ll take the mindset that’s “it is still early days for cloud innovation” and there’s plenty of time to fix these difficult problems.

Feature image courtesy of Shutterstock user Zastolskiy Victor.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises
• Amazon’s DynamoDB: rattling the cloud market

Our site is currently experiencing some issues. Our team is working hard to resolve. Stay tuned.—
  (@LinkedIn) November 05, 2012

#LinkedIn is down or having service trouble, based on user reports/other sources downrightnow.com/linkedin ~a5—
downrightnow (@downrightnowtoo) November 05, 2012

There’s not a lot of (or really any) information available, so we’ll update as needed. This snafu comes on the heels of several service outages — of Google App Engine, of Amazon Web Services, in the past few weeks.

Update: as of 8:43 a.m. PDT, LinkedIn is back online.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• A near-term outlook for big data
• The new IT manager, part 1

Update: Google’s latest update posted at  10:52 pm PDT said the company was working to correct ongoing issues but still sees slow response times and elevated error rates.

“The malfunction appears to be limited to a single component which routes requests from users to the application instance they are using, and does not affect the application instances themselves.”

Update: This posted to the GAE status thread at 9:33 a.m. PDT:

“At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine.  The symptoms that service users would experience include slow response and an inability to connect to services.  We currently show that a majority of App Engine users and services are affected.  Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes.”

The outage was noted all over Twitter and a visit to the GAE status page around noon eastern time turned up an error message. To be fair there appears to be more going on beyond GAE: Both Dropbox and Tumblr have had widespread availability issues today.

Both #Tumblr and Google App Engine, #GAE, are down. What gives?—
Dylan Hurd (@dylanhurd) October 26, 2012

A few minutes later the status page was up again, bearing this message:

Oct 26 2012, 07:30 AM – Oct 26 2012, 11:59 PM
posted by alevi

App Engine is currently experiencing serving issues. The team is actively working on restoring the service to full strength. Please follow this thread for updates:

https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo

BreakingNews.com and our mobile apps are temporarily offline due to a hosting outage at Google. Coverage continues here on @BreakingNews—
Breaking News (@BreakingNews) October 26, 2012

Hey, the Google Apps Engine (#GAE) is down-ish. Millions of apps failing. Keep track of the outage here: code.google.com/status/appengi…—
Matthew A. Peskay (@mpeskay) October 26, 2012

We’ll update this post as we learn more about the nature of the outage, and let us know if you or your business is having problems.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Microsoft Azure: What It Is, What It Costs and Who Should Care
• Cloud and data first-quarter 2013: analysis and outlook

Here’s the lowdown from the  AWS status page:

10:38 AM PDT We are currently investigating degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region.

11:11 AM PDT We can confirm degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. Instances using affected EBS volumes will also experience degraded performance.

11:26 AM PDT We are currently experiencing degraded performance for EBS volumes in a single Availability Zone in the US-EAST-1 Region. New launches for EBS backed instances are failing and instances using affected EBS volumes will experience degraded performance.

12:32 PM PDT We are working on recovering the impacted EBS volumes in a single Availability Zone in the US-EAST-1 Region.

1:02 PM PDT We continue to work to resolve the issue affecting EBS volumes in a single availability zone in the US-EAST-1 region. The AWS Management Console for EC2 indicates which availability zone is impaired.EC2 instances and EBS volumes outside of this availability zone are operating normally. Customers can launch replacement instances in the unaffected availability zones but may experience elevated launch latencies or receive ResourceLimitExceeded errors on their API calls, which are being issued to manage load on the system during recovery. Customers receiving this error can retry failed requests.

2:20 PM PDT We’ve now restored performance for about half of the volumes that experienced issues. Instances that were attached to these recovered volumes are recovering. We’re continuing to work on restoring availability and performance for the volumes that are still degraded.

The site is down right now. It appears to be a network-related issue. We are investigating.—
reddit status (@redditstatus) October 22, 2012

Update: API Offline, Elevated Error Rates status.heroku.com/incidents/456#…—
Heroku Status (@herokustatus) October 22, 2012

You may have noticed that some of your favorite sites are down, including Foursquare. We're hoping things will be back to normal soon!—
foursquare support (@4sqSupport) October 22, 2012

Stay tuned for updates here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A field guide to cloud computing: current trends, future opportunities
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Platform as a Service in 2012