Netflix dives into AWS usage monitoring with Ice

netflixopensource

There is no shortage of tools that measure and monitor Amazon Web Services usage and spending; check out Cloudyn, Cloudability, Newvem, CloudCheckr, Cloud Vertical  et al. And then there is also Amazon’s own Trusted Advisor. But if you want to use a tool that the biggest of the big AWS customers use, you may want to check out Netflix Ice.

Netflix just posted the tool, which provides a birds-eye view of its own cloud landscape (cloudscape?), onto its Github page, the last of a series of open-sourced Netflix goodies to go up.

Asked why Netflix went its own way with an AWS monitoring tool, Ariel Tseitlin, director of cloud solutions for Netflix responded by email to say:

“We built Ice to to give us deep insight into our cloud usage that we couldn’t find with any of Amazon’s or 3rd party offerings.  It gives us the visibility and operational support to manage our mature complex environment and we hope that the rest of the cloud community can benefit from it.”

Like the other AWS checkers, Netflix Ice relies on data supplied by Amazon’s own APIs. With those stats in hand, it tracks usage by accounts; regions; type of service (EC2, S3, EBS); and usage types by instance size. It also tracks whether you use on-demand, reserved or other instance types, the pricing of which varies. Given the sheer number of services and options AWS offers, it’s clear that tracking all of that is a handful, especially for a large organization.

Ice is a Grails project consisting of a processor, a reader and a UI. The processor takes in the Amazon billing file and makes it available to the reader, which renders it to the UI. The UI queries the reader and renders interactive graphs and tables for the browser.

Netflix has years of institutional knowledge on this topic, relying as it does on Amazon infrastructure.  As the blog post states:

Netflix is a highly decentralized environment where each service team decides how many resources their services need.  The elastic nature of the cloud make capacity planning less crucial and teams can simply add resources as needed.  Viewing the broad picture of cloud resource usage becomes more difficult in such an environment.  To address both needs, Netflix created Ice.

netflix-logoSo, if you’re a big AWS shop and would really like to get a handle on just what all your developers have running (or deployed and not running) on AWS cloud, you might want to check Ice out.

Los Gatos, Calif.-based Netflix is really trying to propagate a range of the tools and utilities it uses to make sure its streaming media empire runs well on AWS. Most recently, it posted Isthmus, which vows to manage elastic load balancing across AWS regions. Isthmus, in turn, builds on Zuul, another tool that acts as a gatekeeper between Netflix’ own API and other Netflix services and AWS Elastic Load Balancer that routes video to users. The goal is to prevent meltdowns like the one that hit Netflix last Christmas Eve. 

Netflix, as its cloud guru Adrian Cockroft said at an open source open house it hosted a few months ago, really wants outside companies to deploy its components and, it’s very interested in getting other, non AWS cloud vendors, to deploy these tools as well. One can only imagine why. Cockcroft will be joining us this week at Structure in San Francisco so come by to see what he has to say.

Update: Now that Amazon and Netflix have joined a half dozen or more third-party AWS monitoring tools in the pool, I’d  love to hear from some of the competitors in this space — folks, what does Netflix’ move mean to you and your AWS monitoring tools? Use comments to respond.

This story was updated at 4 p.m. PDT with additional competitors in this space and a request for comments.

loading

Comments have been disabled for this post