6 Comments

Summary:

Very few Amazon Web Services customers know that infrastructure better than Netflix. Now it’s open sourcing its tool for tracking AWS use and spending.

netflixopensource

There is no shortage of tools that measure and monitor Amazon Web Services usage and spending; check out Cloudyn, Cloudability, Newvem, CloudCheckr, Cloud Vertical  et al. And then there is also Amazon’s own Trusted Advisor. But if you want to use a tool that the biggest of the big AWS customers use, you may want to check out Netflix Ice.

Netflix just posted the tool, which provides a birds-eye view of its own cloud landscape (cloudscape?), onto its Github page, the last of a series of open-sourced Netflix goodies to go up.

Asked why Netflix went its own way with an AWS monitoring tool, Ariel Tseitlin, director of cloud solutions for Netflix responded by email to say:

“We built Ice to to give us deep insight into our cloud usage that we couldn’t find with any of Amazon’s or 3rd party offerings.  It gives us the visibility and operational support to manage our mature complex environment and we hope that the rest of the cloud community can benefit from it.”

Like the other AWS checkers, Netflix Ice relies on data supplied by Amazon’s own APIs. With those stats in hand, it tracks usage by accounts; regions; type of service (EC2, S3, EBS); and usage types by instance size. It also tracks whether you use on-demand, reserved or other instance types, the pricing of which varies. Given the sheer number of services and options AWS offers, it’s clear that tracking all of that is a handful, especially for a large organization.

Ice is a Grails project consisting of a processor, a reader and a UI. The processor takes in the Amazon billing file and makes it available to the reader, which renders it to the UI. The UI queries the reader and renders interactive graphs and tables for the browser.

Netflix has years of institutional knowledge on this topic, relying as it does on Amazon infrastructure.  As the blog post states:

Netflix is a highly decentralized environment where each service team decides how many resources their services need.  The elastic nature of the cloud make capacity planning less crucial and teams can simply add resources as needed.  Viewing the broad picture of cloud resource usage becomes more difficult in such an environment.  To address both needs, Netflix created Ice.

netflix-logoSo, if you’re a big AWS shop and would really like to get a handle on just what all your developers have running (or deployed and not running) on AWS cloud, you might want to check Ice out.

Los Gatos, Calif.-based Netflix is really trying to propagate a range of the tools and utilities it uses to make sure its streaming media empire runs well on AWS. Most recently, it posted Isthmus, which vows to manage elastic load balancing across AWS regions. Isthmus, in turn, builds on Zuul, another tool that acts as a gatekeeper between Netflix’ own API and other Netflix services and AWS Elastic Load Balancer that routes video to users. The goal is to prevent meltdowns like the one that hit Netflix last Christmas Eve. 

Netflix, as its cloud guru Adrian Cockroft said at an open source open house it hosted a few months ago, really wants outside companies to deploy its components and, it’s very interested in getting other, non AWS cloud vendors, to deploy these tools as well. One can only imagine why. Cockcroft will be joining us this week at Structure in San Francisco so come by to see what he has to say.

Update: Now that Amazon and Netflix have joined a half dozen or more third-party AWS monitoring tools in the pool, I’d  love to hear from some of the competitors in this space — folks, what does Netflix’ move mean to you and your AWS monitoring tools? Use comments to respond.

This story was updated at 4 p.m. PDT with additional competitors in this space and a request for comments.

  1. Excellent stuff & good to see Netflix’s take on large scale aws deployments & usecases. Attribo has been doing this since long & fits well for a mid-large de centralized dev/ops teams along with viewes for CxO & even finance teams to keep an eye aws consumption.
    Congrats to the Netflix team for pulling this through. Glad to welcome you to attribo dot com & see for yourself.

    Share
  2. This exciting news validates the emerging Cloud Monitoring & Optimization space addressing the pressing need to align AWS consumption with business goals. Still, it’s interesting to see how different solutions for the same business pain-point can happily coexist.

    Ice potentially offers the devops crowd an open source platform they can configure and tweak ad infinitum. Cloudyn focuses on the business manager and c-level audiences who demand a hassle-free, end-to-end solution to optimize their cloud deployment based on actionable recommendations.

    Netflix is doing an excellent job running the largest AWS deployment and integrating it into the core of their business. At that scale, it makes sense to build operational tools that manage such a massive undertaking with dedicated resources to bring tools like Ice to market.

    In contrast, we are getting very consistent feedback from hundreds of enterprise customers. They want to REDUCE cloud spending. From their perspective, integrating an open platform will accomplish just the opposite.

    Share
  3. We think it’s terrific that Netflix is offering Ice to the AWS community. The offering reinforces our core belief for building CloudCheckr: although AWS offers outstanding benefits, these benefits often require specialized tools to fully realize.

    Make no mistake – we agree with Netflix that resource cost reporting is essential. However, rather than simply reporting costs, CloudCheckr (http://bit.ly/11Zfrdv) provides comprehensive usage analysis to enable our users to deploy resources in a way that fully leverage the elasticity of AWS. CloudCheckr not only looks at the resource (as does Ice) but it also analyzes the usage of the resource. The resulting analytics and metrics are then organized to provide detailed reports and utilization heat maps. CloudCheckr also includes sizing and purchasing recommendations. These additional steps are essential to controlling your costs and optimizing your spend over time by deploying exactly what you need.

    We at CloudCheckr would also like to note one other key reality of AWS: cost is not the only issue that needs attention for optimization. CloudCheckr recognizes that optimization also requires usage insight and security. With that in mind, CloudCheckr provides hundreds of best practices that identify issues ranging from misconfigured Auto Scaling groups to open permissioned S3 buckets to expiring SSL certificates to wide IP addresses and open ports. CloudCheckr also provides change monitoring and automated email alerts to insure that users are constantly informed and aware of their cost, usage, and security.

    We have posted a full discussion of our perspective on Ice – including 3 key business issues that every user should consider – here: http://bit.ly/1asvvrB

    If you have not already, we encourage you to take a free 2 week test drive of CloudCheckr Pro here: http://bit.ly/1a4G18r . Just enter ‘Barb’ in the promotion code and we will extend your trial to 4 weeks. Try it for yourself and let us know what you think.

    Share
  4. Thanks, Barb. Netflix’s push to develop these open source tools and make them available to other organizations is very interesting. You could also have mentioned the Simian Army – a collection of tools (with awesome names like Chaos Monkey and Chaos Gorilla) that Netflix uses to test the resilience of its cloud applications. I’d really like to read some analysis of what Netflix’s end game is.

    Share
  5. Another entrant in the Cloud Economics space … I’m not sure how much more ICE does above Trusted Advisor (or what AWS plans for TA to do) + Programmatic Billing do, but visibility into the new variables IaaS deployments create is a good thing. The only caveat about OSS tools is that you have to self-host them, which goes against the everything-as-a-Service model that really underpins the power of the Cloud. I think there’s always a need for expertly created, maintained, supported SaaS tools at all layers of the stack.

    Share
  6. Barb, in response to your request, at Teevity, we’ve been toying aroung with Ice since it’s been released and we’ve made it available to a select group of our existing customers, with interesting feedback (check our blog – http://bit.ly/14W5Wjv).

    Given this positive feedback and the good community around Ice, we have decided to adopt Ice’s code base for a part of our service, leading to two things :
    – A “Detailed AWS costs analytics” feature built on Ice inside the Teevity SaaS service (pricing info coming soon).
    – A “Standalone, run it on your own AWS account” solution, using a CloudFormation (pricing info also coming soon through the AWS Marketplace).

    If you’re interested (free beta for now), you can register here : http://ice.teevity.com/register/

    And for companies who are not willing to use SaaS cloud costs analytics services (because they have strong confidentiality requirements for instance), the standalone option is by far the easiest and most cost-effective way to run Ice today.

    We are already adding features (access protection, easy AWS tags handling, graphical markers to enable correlations between actions and their impact on costs, …), some of which will be contributed back to the community (check our Github account).

    Share

Comments have been disabled for this post