I decided to get my geek-on Wednesday night and attend the first Netflix Open Source Software meet-up. I’m glad I did. I learned more about the challenges of running at scale in the cloud (or at least in Amazon’s cloud) at this event than I have at the past few conferences I’ve been to, including AWS Re:Invent (which was mostly hype).
Adrian Cockcroft, chief cloud guru at Netflix ran through the top OSS projects Netflix is working on to make its movie streaming service run better on AWS. These projects essentially amount to a software framework or platform that others can use to support the deployment of their own apps on AWS, and eventually, other clouds. Here are the lightening talks about each OSS project and how they fix issues Netflix has experienced using AWS.
A number of questions spring to mind. This collection of OSS tools is starting to look like an open source PaaS framework, although it doesn’t sound like Netflix has plans to become a PaaS provider, at least not at the moment. I wonder if these cloud management tools are so optimized for Netflix that it would take a lot of adapting to make them more generically useful to other apps running on AWS? In other words, the individual tools (Eureka, Asgard, Edda, Hystrix, Odin etc) might be helpful, but the idea that all of these stitched together form a platform for anyone to use, seems like a leap.
In the same vein, Netflix promised that its OSS tools will be more portable in the future, potentially working across other clouds. But it seemed like many of the projects were designed to fix issues specifically with Amazon EC2, so I’m wondering how these tools might be applicable to other cloud infrastructures.
Cockcroft explained that one of the reasons Netflix is putting so much work into building these tools and open sourcing them in the hopes others will use them, is to avoid the potential switching costs of having to use other tools that might become more popular. Hear that RightScale, Enstratus?
He used the analogy of the pioneers moving west, each wagon forging its own path, until eventually someone laid down a road, making it easier for everyone to get there. It would be a helpful analogy, if only everyone using the cloud had the same goal or use case in mind. Far from it. Just about everyone we talk to has a different set of parameters and requirements from their cloud provider. For example, there isn’t a Netflix OSS project that focuses on how to treat compliance data, a key requirement for enterprises considering cloud services. Pharmaceutical drug development is a very different cloud use case than Walmart using the cloud for point of sale data analysis. There’s an incredible amount of variety in how we use computing, moving it to the cloud doesn’t change that.
My suspicion is that there will never be one road to the cloud, but rather many equally well paved, but different routes, of which the Netflix OSS tools will be just one, optimized for video streaming services at that.