Comments on: Can Today's Hardware Handle the Cloud?

By: SpringSource Buys Startup to Scale Messaging in the Cloud

SpringSource Buys Startup to Scale Messaging in the Cloud — Wed, 11 Aug 2010 00:07:15 +0000

[...] backed by major banks, Cisco and a handful of smaller companies. As hardware is virtualized, translating some of the network equipment like load balancers into software allow services running on the virtualized hardware to scale better. Hopefully we’ll learn [...]

By: Will Microsoft Tempt Enterprises Up To the Cloud? - GigaOM

Will Microsoft Tempt Enterprises Up To the Cloud? - GigaOM — Tue, 28 Oct 2008 20:28:59 +0000

[...] very sensitive data, and all of them said their clients would balk at cloud storage until they get a closer look at the security and reliability of the architecture. In the U.S. there are, at the very least, regulatory hurdles around storing sensitive data [...]

By: Scale Fail : Beyond Search

Scale Fail : Beyond Search — Mon, 21 Jul 2008 13:11:12 +0000

[...] shifting online, web services are still fragile, in part because we are still using technologies built for a much less strenuous [...]

By: S3 Outage Highlights Fragility of Web Services - GigaOM

S3 Outage Highlights Fragility of Web Services - GigaOM — Mon, 21 Jul 2008 03:10:46 +0000

[...] shifting online, web services are still fragile, in part because we are still using technologies built for a much less strenuous [...]

By: Craig Balding

Craig Balding — Thu, 03 Jul 2008 08:30:33 +0000

Alistair

My take on this is similar to that of Fazal. Amazon support API level integrity checks in the form of MD5 to detect potential corruption at time of transfer. It was the people using MD5 that picked up this issue – for others their data was getting silently corrupted (!).

As S3 is primarily targeted at developers, this incident demonstrates the need for greater awareness around Cloud Storage integrity API options and limitations.

I’ve posted on the issue here:
http://cloudsecurity.org/2008/06/25/a-question-of-integrity-to-md5-or-not-to-md5/

Thanks,
Craig

By: David Ulevitch

David Ulevitch — Fri, 27 Jun 2008 21:52:01 +0000

1) The network is designed to scale. A broken loadbalancer doesn’t indicate anything.
2) The customers in the clouds are buying Arastra switches, the exact same hardware base Google is using (Google just using their own software). All these chips are made by Fulcrum Microsystems. They are doing *very* well.
3) This is nothing new. It is not a threat to Cisco or Juniper. Cisco will eventually just buy Arastra. This happens every couple years.

By: Fazal Majid

Fazal Majid — Fri, 27 Jun 2008 18:59:26 +0000

I think Amazon handled this one quite well, in fact.

The error was introduced in the load balancer in the part where it copies data from one incoming SSL connection to an outgoing (presumably non-SSL) connection. This could be due to any number of reasons, including defective memory or firmware code bugs. Since the corruption was introduced between connections, the TCP checksum mechanisms on either connection wouldn’t have caught it, which is why there is still a need for end-to-end checksums. Amazon’s API does actually have such a mechanism, it was just not required or enforced inconsistently. I am sure they will fix that in the next release.

The conclusion I draw from this incident is the opposite of yours – hardware is much more reliable than software, but you can’t trust it entirely either. Simply replacing the load-balancer with another from a different brand will not eliminate the vulnerability, and thus it is not a solution. Cost or manageability, or defect rates would be good reasons for Amazon to switch load-balancer suppliers.

HP ProCurve has an interesting pitch – all their Ethernet switches now have a programmable CPU core per port, and they offer SDKs to partners to implement custom advanced functionality in those. A ProCurve switch costs an order of magnitude less than a F5 or Netscaler, and the per port ASICs should be perfectly up to the task of load-balancing and simple firewalling (if not SSL acceleration, which is much more computationally intensive). A company like Amazon or Google could save a lot of money by entering the ProCurve partner program and writing their own custom cloud-oriented logic that does exactly what they need and not one bit more, to reduce costs, complexity and the likelihood of bugs as the previous poster noted.

By: Andrew

Andrew — Fri, 27 Jun 2008 16:43:12 +0000

A big part of the problem is a lack of experienced individuals when it comes to operating large-scale clusters. To use the Amazon S3 example above, best practices for extremely large hardware clusters is to use end-to-end software checksums to catch the occasional hardware failure that will allow corruption to get past hardware CRC and checksum systems. It looks like they learned that lesson the hard way, but they should not feel too bad because Google did too. If you put enough silicon in a room, you can no longer count on its internal error correction mechanism and software checks need to be instituted, something originally discovered by the supercomputing community that still has not penetrated the broader developer space.

I will generally agree, though, that a lot of network gear is poorly designed for large-scale distributed systems, either having inexpensive silicon that is too under-powered architecturally for clusters and max-load usage (for the cost conscious markets) or being “carrier class” networking gear with performant silicon but also a ton of other features glued on that are useless for distributed cluster applications and which drive the price way up. It is just a matter of time before one of the networking gear companies starts producing switch engines specifically designed for large-scale cluster applications if they haven’t already.