Comments on: Why Amazon Went Down, and Why It Matters http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/ Trusted Insights and Conversations on the Next Wave of Technology Thu, 26 Nov 2009 15:23:00 +0000 http://wordpress.com/ hourly 1 By: Amazon Was Out For The Count « Web News and Practical websites http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-963872 Amazon Was Out For The Count « Web News and Practical websites Sun, 02 Aug 2009 14:57:55 +0000 http://gigaom.com/?p=13706#comment-963872 <p>[...] So what exactly happened. Well here are the facts from Gigaom [...]</p> [...] So what exactly happened. Well here are the facts from Gigaom [...]

]]> By: Can Today’s Hardware Handle the Cloud? - GigaOM http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-886050 Can Today’s Hardware Handle the Cloud? - GigaOM Fri, 27 Jun 2008 15:48:32 +0000 http://gigaom.com/?p=13706#comment-886050 <p>[...] isn’t the first time load balancers have been implicated in an outage at Amazon. At O’Reilly’s Velocity conference, conference co-chair Jesse Robbins talked about a [...]</p> [...] isn’t the first time load balancers have been implicated in an outage at Amazon. At O’Reilly’s Velocity conference, conference co-chair Jesse Robbins talked about a [...]

]]> By: Milly Dawson http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-884514 Milly Dawson Thu, 19 Jun 2008 19:03:04 +0000 http://gigaom.com/?p=13706#comment-884514 <p>It's too bad about Amazon going down for a spell but maybe it inspired someone or several someones to check out their local library. I hope so.</p> It’s too bad about Amazon going down for a spell but maybe it inspired someone or several someones to check out their local library. I hope so.

]]> By: William Nortman http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883622 William Nortman Fri, 13 Jun 2008 18:14:23 +0000 http://gigaom.com/?p=13706#comment-883622 <p>So how do you manage and diagnose such complex systems? You are probably talking about 1,000s of devices that all could be the root cause of the issue. How do you isolate the root cause? Classically, you’d have some type of monitor with rules to detect when certain issues occurred. This MIGHT point you to the right location. However, with IT being the main differentiator to the end customer, new changes are constantly being rolled out. The system is always growing, changing and the customer usage patterns are always altering over time. So the rules originally written tend to get lightened up so that you don’t have alert storms. Now with the rules loosened up, you might not detect the failure and, even if you do, the events will not be as helpful in detecting the root cause. You could throw more and more smart people at the problem and constantly update and maintain your set of rules. However, as the system gets more and more complex this human cost will grow at an enormous rate. You need something different, a tool that automatically detects issues and adjusts to the changing system and usage patterns. You need a tool that uses statistical analytics to weed through the noise of the system and determines the relationship between the business information and the IT information in order to allow you to quickly get to the root cause of issues like this. If you got a single alert that told you that the load balances were getting a higher than average reconnect rate, your number of sales was dropping below normal, your average load on all web servers was way below normal, the number of search transactions was way below normal, the normal of connected users was way below normal etc., you would be able to have a quick idea on where to start. No need to experience all your line of defenses being down again.</p> So how do you manage and diagnose such complex systems? You are probably talking about 1,000s of devices that all could be the root cause of the issue. How do you isolate the root cause? Classically, you’d have some type of monitor with rules to detect when certain issues occurred. This MIGHT point you to the right location. However, with IT being the main differentiator to the end customer, new changes are constantly being rolled out. The system is always growing, changing and the customer usage patterns are always altering over time. So the rules originally written tend to get lightened up so that you don’t have alert storms. Now with the rules loosened up, you might not detect the failure and, even if you do, the events will not be as helpful in detecting the root cause. You could throw more and more smart people at the problem and constantly update and maintain your set of rules. However, as the system gets more and more complex this human cost will grow at an enormous rate. You need something different, a tool that automatically detects issues and adjusts to the changing system and usage patterns. You need a tool that uses statistical analytics to weed through the noise of the system and determines the relationship between the business information and the IT information in order to allow you to quickly get to the root cause of issues like this. If you got a single alert that told you that the load balances were getting a higher than average reconnect rate, your number of sales was dropping below normal, your average load on all web servers was way below normal, the number of search transactions was way below normal, the normal of connected users was way below normal etc., you would be able to have a quick idea on where to start. No need to experience all your line of defenses being down again.

]]> By: Links List 6.13.08 | IT's About Uptime - The StackSafe Blog http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883583 Links List 6.13.08 | IT's About Uptime - The StackSafe Blog Fri, 13 Jun 2008 14:49:46 +0000 http://gigaom.com/?p=13706#comment-883583 <p>[...] light of Amazon’s latest downtime issues, Gigaom explains why Amazon went down and why it matters. In a thorough explanation, Gigaom bets the problem to be with the CDN or AFE. The moral of their [...]</p> [...] light of Amazon’s latest downtime issues, Gigaom explains why Amazon went down and why it matters. In a thorough explanation, Gigaom bets the problem to be with the CDN or AFE. The moral of their [...]

]]> By: A Tale of Two Outages | IT's About Uptime - The StackSafe Blog http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883246 A Tale of Two Outages | IT's About Uptime - The StackSafe Blog Wed, 11 Jun 2008 16:20:39 +0000 http://gigaom.com/?p=13706#comment-883246 <p>[...] downtime incidents crossed our paths recently that we thought deserved comment. You probably have read about the first (Amazon), but maybe missed the second (Southern Company Nuclear Power [...]</p> [...] downtime incidents crossed our paths recently that we thought deserved comment. You probably have read about the first (Amazon), but maybe missed the second (Southern Company Nuclear Power [...]

]]> By: Vito http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883231 Vito Wed, 11 Jun 2008 14:37:41 +0000 http://gigaom.com/?p=13706#comment-883231 <p>I still have the headers in my scroll back buffer:</p> <p>$ wget -S http://www.amazon.com -O /dev/null --14:22:58-- http://www.amazon.com/ => `/dev/null' Resolving www.amazon.com... 72.21.210.11 Connecting to www.amazon.com|72.21.210.11|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 503 Service Unavailable Server: NS_6.1 Content-Length:62 Connection: close 14:22:58 ERROR 503: Service Unavailable.</p> <p>It appears that they are indeed running Citrix Netscalers (Server: NS_6.1) which is what returned the 503 error you see above.</p> <p>"Sorry" pages only work if configured; they are not a default. Maybe Amazon hasn't gotten around to that. ;)</p> I still have the headers in my scroll back buffer:

$ wget -S http://www.amazon.com -O /dev/null –14:22:58– http://www.amazon.com/ => `/dev/null’ Resolving http://www.amazon.com... 72.21.210.11 Connecting to http://www.amazon.com|72.21.210.11|:80… connected. HTTP request sent, awaiting response… HTTP/1.1 503 Service Unavailable Server: NS_6.1 Content-Length:62 Connection: close 14:22:58 ERROR 503: Service Unavailable.

It appears that they are indeed running Citrix Netscalers (Server: NS_6.1) which is what returned the 503 error you see above.

“Sorry” pages only work if configured; they are not a default. Maybe Amazon hasn’t gotten around to that. ;)

]]> By: Fortresses in the Clouds: On-demand Platforms Had Better Build Moats - GigaOM http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883103 Fortresses in the Clouds: On-demand Platforms Had Better Build Moats - GigaOM Tue, 10 Jun 2008 23:18:03 +0000 http://gigaom.com/?p=13706#comment-883103 <p>[...] Friday, Amazon’s U.S. site went off the air, and later some of its other properties were unavailable. Lots of folks who wouldn’t let me quote [...]</p> [...] Friday, Amazon’s U.S. site went off the air, and later some of its other properties were unavailable. Lots of folks who wouldn’t let me quote [...]

]]> By: michaelportent http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-883086 michaelportent Tue, 10 Jun 2008 21:02:47 +0000 http://gigaom.com/?p=13706#comment-883086 <p>@Mel: I think that rumor is hilarious too. It wouldn't surprise me that a bunch of gamers writing scripts to auto-buy available items could crash Amazon. That would be awesome.</p> @Mel: I think that rumor is hilarious too. It wouldn’t surprise me that a bunch of gamers writing scripts to auto-buy available items could crash Amazon. That would be awesome.

]]> By: Michael Janke http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882997 Michael Janke Tue, 10 Jun 2008 12:41:35 +0000 http://gigaom.com/?p=13706#comment-882997 <p>@john adams: Netscaler load balancers (as Amazon is rumored to use) can re-direct to a 'Sorry Page' when all backend services are down. It's a simple config, and we require it on all our load balanced services.</p> <p>That, and the fact that the HTTP error page that was presented looks just like the one generated by Netscalers when operating in proxy mode, indicates to me that the load balancer layer was up & functional, but there was nothing behind it to which to send traffic, and that there was no redirect enabled.</p> <p>Having said that, re-directing a high volume site to a sorry page is a challenge itself. We maintain a load balanced pool of servers on a separate pair of load balancers, just to handle the sorry page from the load balanced applications.</p> <p>--Mike</p> @john adams: Netscaler load balancers (as Amazon is rumored to use) can re-direct to a ‘Sorry Page’ when all backend services are down. It’s a simple config, and we require it on all our load balanced services.

That, and the fact that the HTTP error page that was presented looks just like the one generated by Netscalers when operating in proxy mode, indicates to me that the load balancer layer was up & functional, but there was nothing behind it to which to send traffic, and that there was no redirect enabled.

Having said that, re-directing a high volume site to a sorry page is a challenge itself. We maintain a load balanced pool of servers on a separate pair of load balancers, just to handle the sorry page from the load balanced applications.

–Mike

]]> By: the application delivery network » Load balancers don't have what??? http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882903 the application delivery network » Load balancers don't have what??? Mon, 09 Jun 2008 19:45:37 +0000 http://gigaom.com/?p=13706#comment-882903 <p>[...] have what??? Posted by: The ADC in Events, Opinion Alistair Croll over at Gigaom had an interesting dissection of Amazon’s recent outage and an interesting deduction based on the facts regarding what went [...]</p> [...] have what??? Posted by: The ADC in Events, Opinion Alistair Croll over at Gigaom had an interesting dissection of Amazon’s recent outage and an interesting deduction based on the facts regarding what went [...]

]]> By: BloggerBen http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882897 BloggerBen Mon, 09 Jun 2008 19:31:36 +0000 http://gigaom.com/?p=13706#comment-882897 <p>This may be a dumb question, but did the Amazon cloud go down too?</p> This may be a dumb question, but did the Amazon cloud go down too?

]]> By: John Franks http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882893 John Franks Mon, 09 Jun 2008 19:05:44 +0000 http://gigaom.com/?p=13706#comment-882893 <p>Check out David Scott's interview at the Business Forum: http://www.businessforum.com/DScott_02.html. It seems Amazon (and a lot of other organizations suffering data thefts, outages, bad projects, etc.) needs his book!</p> Check out David Scott’s interview at the Business Forum: http://www.businessforum.com/DScott_02.html. It seems Amazon (and a lot of other organizations suffering data thefts, outages, bad projects, etc.) needs his book!

]]> By: Lori MacVittie http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882892 Lori MacVittie Mon, 09 Jun 2008 19:01:33 +0000 http://gigaom.com/?p=13706#comment-882892 <p>@John Adams</p> <p>I think you haven't been looking around very much. Modern load balancers (application delivery controllers) a la F5 BIG-IP, have configurable "apology" pages when all nodes are down. This technology has been around for quite a while, it's not something new or unknown in the industry at all.</p> <p>Lori</p> @John Adams

I think you haven’t been looking around very much. Modern load balancers (application delivery controllers) a la F5 BIG-IP, have configurable “apology” pages when all nodes are down. This technology has been around for quite a while, it’s not something new or unknown in the industry at all.

Lori

]]> By: Avneesh Balyan http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882879 Avneesh Balyan Mon, 09 Jun 2008 17:21:37 +0000 http://gigaom.com/?p=13706#comment-882879 <p>Seems like, The site is down again....</p> <p>I was thinking of Amazon as Google in Shopping domain (item search, reviews etc..). Time to re-tink???</p> Seems like, The site is down again….

I was thinking of Amazon as Google in Shopping domain (item search, reviews etc..). Time to re-tink???

]]> By: robotthink http://gigaom.com/2008/06/06/why-amazon-went-down-and-what-it-means-to-you/#comment-882737 robotthink Sun, 08 Jun 2008 21:28:44 +0000 http://gigaom.com/?p=13706#comment-882737 <p>Alastair and Geo,</p> <p>It most certainly was a DOS attack, I assure you.</p> <p>And Seattle-ite is right about everything he says in his post, and that was in part why the DOS was possible.</p> Alastair and Geo,

It most certainly was a DOS attack, I assure you.

And Seattle-ite is right about everything he says in his post, and that was in part why the DOS was possible.

]]>