9 Comments

Summary:

Amazon Web Services has upped the maximum image size in Amazon Simple Storage Service (S3) to 5TB from 5GB. Clearly, a 1,000X increase is significant and probably is driven by the new class of customers AWS is catering to with its Cluster Compute and GPU Instances.

movies stack

Amazon CTO Werner Vogels announced on his blog this morning that Amazon Web Services has upped the maximum object size in the Amazon Simple Storage Service (S3) to from 5GB to 5TB. Clearly, a 1,000x increase per object is significant and, as Vogels alludes, is driven by the new class of customers AWS is catering to with its Cluster Compute and GPU Instances, who need serious cloud storage capabilities as well as serious cloud computing capabilities.

As Vogels notes, researchers and movie studios can easily produce objects that surpass the previous 5GB limit: “a 2-hour movie on Blu-ray can be 50 gigabytes. The same movie stored in an uncompressed 1080p HD format is around 1.5 terabytes.” He doesn’t mention Netflix by name, but I have to assume the idea of getting even more of its business has the Amazon financial team salivating. There are only so many scientists running HPC workloads and many movie studios have their own clusters (or use services like Weta Digital), but there are countless movies for Netflix to store, and enough business to justify doing so. Netflix is currently using Amazon S3 and EC2 for a variety of tasks, although it hasn’t cited HD video storage as one of them. Perhaps the boost in object size will change this.

The advent of 5TB files also helps explain the presence of AWS’s Multipart Upload service, which the company rolled out last month. Vogels explains it in more detail, as does the original blog post about it, but the gist is that users can break files in as many as 1,024 segments, ensuring that a network failure won’t kill an entire upload. Anyone who has backed up a hard drive in the cloud knows how long that upload can take;  imagine a 5TB file, or a library of them.

As yesterday’s release of Android and iOS software development kits illustrates, AWS can’t afford to give up wooing developers, but this huge storage upgrade is further evidence that Amazon has its eyes set on winning bigger customers with bigger computing needs. I’ve said in the past that AWS needs a PaaS offering, but I’m beginning to question just how much it does. It doesn’t need to rely on developers if it can tap into huge amounts of infrastructure spending by high-performance users.

Image courtesy of Flickr user makelessnoise.

Related content from GigaOM Pro (sub req’d):

  1. When it comes to the uncompressed 1080p movies, I don’t think Netflix was the customer in mind. With digital movie theaters are becoming more and more common (a new 16-screen 100% digital theater just opened near my house 2 weeks ago), it is going to require a new method of distributing this huge amount of content. As I mentioned, the new theater near my house is all digital… most of the projectors are 1080p, but I believe I heard one was 4K… if a 1920×1080 or just about 2K picture takes up 1.5TB, how much will a 4K, uncompressed movie take up? As mentioned in the article, there is a need to break this up into segments in case there is an error, it can recover more easily. Movie studios would find it much easier to let content storage and delivery guys like AWS be their distribution provider. Instead of the physical shipping using some currier service to move film out to the theaters, they would be using something like this. With all the hoopla we’ve heard lately with Level 3 and Comcast over delivering Netflix, can you imagine the backlash we’d be hearing if people where attempting to stream 1.5TB movies to their house using their own Internet service provider?!?! Movie theaters would likely have dedicated multi-gigabit fiber to Amazon instead of relying on a local Internet provider. It would also make sense for the movie studios to have their own dedicated lines, making Amazon just the clearing house distributer. I really doubt Netflix is as big a player as the article seems to indicate.

    Share
  2. Derrick Harris Friday, December 10, 2010

    The movie theater idea is very interesting, especially if they could get dedicated delivery lines. But I think Netflix still has a use for storing HD videos, even if they’re not delivering uncompressed 1080p versions.

    http://news.cnet.com/8301-27076_3-20000054-248.html

    Share
  3. [...] News thread on Werner’s post pretty interesting, so I though I’d share. Of course, I have my own take, which involves providing an ideal place for companies like Netflix to house HD [...]

    Share
  4. Amazon had to do it for its own video store. They have to be competing with Netflix.

    Share
  5. I think correlating Netflix kind of a scenario with the Amazon cluster instances makes a lot of sense. It is a perfect scenario if we were to consider the GPU Clusters nodes being used for heavy HD Video transcoding jobs from raw formats, and the finished output of which have to be moved to a durable storage that has enough capacity to store such rendered /finished files.

    Also,with the way things are evolving in the field of Cloud computing and particularly at Amazon and specifically services like the CloudFront, I personally think, it may not be a long time before Amazon becomes a formidable CDN similar to services like Edgecast,Lime Light or Akamai and when this time does come, I foresee that Amazon may perhaps offer a 10 or 25 TB S3 storage from where HD videos may be seamlessly delivered on their new type of CloudFront CDNs.

    Essentially, what we are seeing today may only a tip of the iceberg.. .

    Sankar Nagarajan

    Share
  6. 5TB is also perfect for storing backup dumps of corporate relational databases. Assuming the company trusts their data to be stored in the cloud, it would provide an ideal, easy to implement offsite disaster recovery model.

    Share
  7. [...] even more storage capacity. As Amazon CTO Werner Vogels noted when Amazon Web Services said that it now supports 5TB objects in Amazon S3, HD video files can be terabytes apiece. Assuming the list of services streaming consumer content [...]

    Share
  8. [...] All that said, the two also have a somewhat symbiotic relationship, as Netflix is a huge Amazon Web Services (AWS) customer. Since announcing that it was moving to the cloud last year, Netflix has increasingly relied on AWS to power the backend infrastructure behind its streaming business, and may be one reason why Amazon recently increased file object size for its Simple Storage Service. [...]

    Share
  9. [...] that occurs when transferring data between EC2 and S3 servers. This will be especially handy now that AWS supports 5TB objects in S3. Further, Aspera has an on-demand version of its product that utilizes the Amazon infrastructure [...]

    Share

Comments have been disabled for this post