GigaOM » Chaos Monkey

Netflix open sources tool to clean up your AWS cloud

Barb Darrow — Fri, 04 Jan 2013 17:09:17 +0000

Those Netflix engineers are at it again — releasing source code to an in-house tool they use to keep their Amazon Web Services pared down and tidy. Janitor Monkey automates the detection and clean up of unused AWS resources. The advantage of public cloud infrastructure is it’s so easy to spin up new compute instances and add storage willy nilly but not so easy to keep track of those resources as they fall into disuse. That leads to a profusion of under-utilized or totally unused resources — the untidy closet aspect of cloud computing.

Before, Netflix engineers could use its Asgard cloud deployment tool to manually delete these unused resources but automation is key here — especially for big, spread-out workloads.

Janitor Monkey is the latest member of Netflix’s Simian Army tool set which also includes Chaos Monkey. Open sourced in July, Chaos Monkey shuts down computing instances in a controlled way to ensure that applications keep running even when a virtual server goes down unexpectedly.

Netflix is famously one of AWS’ biggest users and has tons of savvy about how to make the most of those services — which is why it’s ironic that Netflix itself has been brought low by problems at the AWS US-East data center as evidenced in the great Christmas Eve on-demand video meltdown of 2012.

Feature photo courtesy of Flickr user Robert S. Donovan

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How NASA battle-tested its Mars rover live stream

Derrick Harris — Sun, 05 Aug 2012 17:18:20 +0000

Updated: Anyone excited to watch NASA’s Mars rover, called Curiosity, land on the surface of the red planet on Sunday night can all but rest assured that too much demand won’t kill the stream. NASA teamed with an application-testing specialist called SOASTA to ensure the world can keep watching even if demand spikes or servers fail, proving a single implementation of its application stack can handle 25 Gigabits per second of web traffic.

SOASTA tests the traffic load applications can handle by generating cloud-computing-based resources that mimic the traffic generated by potentially millions of simultaneous real-world users. The company also recently tested London2012.com, the official Olympics web site that organizers predict will have to handle more than a billion visits over the course of this year’s event.

According to an e-mail explanation sent to me by NASA and SOASTA, here’s how the two groups put Curiosity’s streaming infrastructure, which is hosted on the Amazon Web Services cloud, through its paces:

They built a test infrastructure comprised of a single origin server (a Mac Pro housed at NASA’s Jet Propulsion Laboratory) serving four bitrates (250, 500, 750 and 1,000) to a single Flash Media Server. Output was cached by a single “tier 1″ Nginx server, fronted by 40 “tier 2″ load-balanced Nginx servers running on Amazon EC2.
SOASTA generated load from six Amazon EC2 regions across the world, generating more than 25 Gbps of traffic and pounding the application for nearly 40 minutes.
After 20 minutes, they terminated 10 instances (see Arrow 1 on the chart) to see if their stack and Amazon’s cloud could handle the failure. This temporarily reduced the amount of traffic the system could handle, but Amazon’s Elastic Load Balancer service had the failed instances back up and handling 25 Gbps in about 5 minutes.
When the team terminated 20 instances (see Arrow 3), the remaining servers’ traffic-handling rate dropped to 12 Gbps and servers started showing signs of being overloaded. Once again, Elastic Load Balancer brought the instances back up (see Arrow 4) and the traffic rate returned to its initial 25 Gbps.
All told, SOASTA’s load-testing servers downloaded 68TB of video (see Arrow 2) from NASA’s cache during the nearly 40-minute test.

In the end, the team concluded:

Load on the primary FMS server and the tier 1 cache remained very low for the entirety of the test; we should have no problem running dozens of stacks during the live event. Anecdotal evaluation of the NASA live stream during testing showed no buffering or bitrate drops.

We are confident that the results of this test suggest that an aggregate of these stacks will be able to deliver the required streaming delivery for the Curiosity landing event.

Overall cost and flexibility benefits aside, the ability to test the effectiveness of an application’s infrastructure relatively easily and inexpensively is turning out to be one of the big benefits of cloud computing. NASA’s Curiosity test is just the latest example of this. Video-rental giant Netflix has built an army of simian-named services (such as Chaos Monkey) that simulate everything from the failure of a single server to the failure of an Availability Zone in Amazon’s cloud, where Netflix runs almost all of its IT operations.

Tune in tonight at 10:31 p.m. Pacific Daylight Time to see if NASA’s Curiosity streaming infrastructure really can hold up.

Update: Amazon Web Services has posted a blog detailing NASA’s production architecture for the Curiosity live stream. You can read the details there, but here’s a diagram of the architecture that shows how the test architecture scaled:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Netflix open sources cloud-testing Chaos Monkey

Derrick Harris — Mon, 30 Jul 2012 16:03:49 +0000

Netflix has a gift for anybody who needs to ensure their cloud-hosted applications keep running even if some of the virtual servers on which they’re running die. It’s called a Chaos Monkey — but don’t worry, this monkey is very tameable and is now open source.

The video rental and streaming giant is one of the world’s biggest consumer of cloud computing resources — it hosts the majority of its infrastructure on the Amazon Web Services cloud — and Netflix developed Chaos Monkey as a method for ensuring that its system is capable of healing itself or continuing to run should instances fail. “Over the last year,” Netflix cloud engineers Cory Bennett and Ariel Tseitlin wrote in a blog post announcing the open source version, “Chaos Monkey has terminated over 65,000 instances running in our production and testing environments. Most of the time nobody notices, but we continue to find surprises caused by Chaos Monkey which allows us to isolate and resolve them so they don’t happen again.”

Anyone scared releasing such a wild-sounding entity into their application infrastructure (or envious that they can’t do so because they don’t run on Amazon’s cloud) need not worry. As Bennett and Tseitlin explain, Chaos Monkey is configurable and “by default, runs on non-holiday weekdays between 9am and 3pm.” It’s also flexible enough to run on clouds other than AWS, they write.

Oh, and Chaos Monkey is just the first of Netflix’s Simian Army to find its way into the open source world. “The next likely candidate will be Janitor Monkey which helps keep your environment tidy and your costs down,” Bennett and Tseitlin note.

Another member of the army, Chaos Gorilla — which is designed to simulate the loss of an entire AWS Availability Zone — recently made headlines when a cascading bug took down part of Amazon’s cloud in late June.

Image courtesy of Shutterstock user Karen Gentry.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.