Blog Post

Is AWS Targeting Netflix With 5TB Objects?

Amazon (s amzn) CTO Werner Vogels announced on his blog this morning that Amazon Web Services has upped the maximum object size in the Amazon Simple Storage Service (S3) to from 5GB to 5TB. Clearly, a 1,000x increase per object is significant and, as Vogels alludes, is driven by the new class of customers AWS is catering to with its Cluster Compute and GPU Instances, who need serious cloud storage capabilities as well as serious cloud computing capabilities.

As Vogels notes, researchers and movie studios can easily produce objects that surpass the previous 5GB limit: “a 2-hour movie on Blu-ray can be 50 gigabytes. The same movie stored in an uncompressed 1080p HD format is around 1.5 terabytes.” He doesn’t mention Netflix (s nflx) by name, but I have to assume the idea of getting even more of its business has the Amazon financial team salivating. There are only so many scientists running HPC workloads and many movie studios have their own clusters (or use services like Weta Digital), but there are countless movies for Netflix to store, and enough business to justify doing so. Netflix is currently using Amazon S3 and EC2 for a variety of tasks, although it hasn’t cited HD video storage as one of them. Perhaps the boost in object size will change this.

The advent of 5TB files also helps explain the presence of AWS’s Multipart Upload service, which the company rolled out last month. Vogels explains it in more detail, as does the original blog post about it, but the gist is that users can break files in as many as 1,024 segments, ensuring that a network failure won’t kill an entire upload. Anyone who has backed up a hard drive in the cloud knows how long that upload can take;  imagine a 5TB file, or a library of them.

As yesterday’s release of Android (s goog) and iOS (s aapl) software development kits illustrates, AWS can’t afford to give up wooing developers, but this huge storage upgrade is further evidence that Amazon has its eyes set on winning bigger customers with bigger computing needs. I’ve said in the past that AWS needs a PaaS offering, but I’m beginning to question just how much it does. It doesn’t need to rely on developers if it can tap into huge amounts of infrastructure spending by high-performance users.

Image courtesy of Flickr user makelessnoise.

Related content from GigaOM Pro (sub req’d):

9 Responses to “Is AWS Targeting Netflix With 5TB Objects?”

  1. 5TB is also perfect for storing backup dumps of corporate relational databases. Assuming the company trusts their data to be stored in the cloud, it would provide an ideal, easy to implement offsite disaster recovery model.

  2. I think correlating Netflix kind of a scenario with the Amazon cluster instances makes a lot of sense. It is a perfect scenario if we were to consider the GPU Clusters nodes being used for heavy HD Video transcoding jobs from raw formats, and the finished output of which have to be moved to a durable storage that has enough capacity to store such rendered /finished files.

    Also,with the way things are evolving in the field of Cloud computing and particularly at Amazon and specifically services like the CloudFront, I personally think, it may not be a long time before Amazon becomes a formidable CDN similar to services like Edgecast,Lime Light or Akamai and when this time does come, I foresee that Amazon may perhaps offer a 10 or 25 TB S3 storage from where HD videos may be seamlessly delivered on their new type of CloudFront CDNs.

    Essentially, what we are seeing today may only a tip of the iceberg.. .

    Sankar Nagarajan

  3. When it comes to the uncompressed 1080p movies, I don’t think Netflix was the customer in mind. With digital movie theaters are becoming more and more common (a new 16-screen 100% digital theater just opened near my house 2 weeks ago), it is going to require a new method of distributing this huge amount of content. As I mentioned, the new theater near my house is all digital… most of the projectors are 1080p, but I believe I heard one was 4K… if a 1920×1080 or just about 2K picture takes up 1.5TB, how much will a 4K, uncompressed movie take up? As mentioned in the article, there is a need to break this up into segments in case there is an error, it can recover more easily. Movie studios would find it much easier to let content storage and delivery guys like AWS be their distribution provider. Instead of the physical shipping using some currier service to move film out to the theaters, they would be using something like this. With all the hoopla we’ve heard lately with Level 3 and Comcast over delivering Netflix, can you imagine the backlash we’d be hearing if people where attempting to stream 1.5TB movies to their house using their own Internet service provider?!?! Movie theaters would likely have dedicated multi-gigabit fiber to Amazon instead of relying on a local Internet provider. It would also make sense for the movie studios to have their own dedicated lines, making Amazon just the clearing house distributer. I really doubt Netflix is as big a player as the article seems to indicate.