Amazon CTO Werner Vogels announced on his blog this morning that Amazon Web Services has upped the maximum object size in the Amazon Simple Storage Service (S3) to from 5GB to 5TB. Clearly, a 1,000x increase per object is significant and, as Vogels alludes, is driven by the new class of customers AWS is catering to with its Cluster Compute and GPU Instances, who need serious cloud storage capabilities as well as serious cloud computing capabilities.
As Vogels notes, researchers and movie studios can easily produce objects that surpass the previous 5GB limit: “a 2-hour movie on Blu-ray can be 50 gigabytes. The same movie stored in an uncompressed 1080p HD format is around 1.5 terabytes.” He doesn’t mention Netflix by name, but I have to assume the idea of getting even more of its business has the Amazon financial team salivating. There are only so many scientists running HPC workloads and many movie studios have their own clusters (or use services like Weta Digital), but there are countless movies for Netflix to store, and enough business to justify doing so. Netflix is currently using Amazon S3 and EC2 for a variety of tasks, although it hasn’t cited HD video storage as one of them. Perhaps the boost in object size will change this.
The advent of 5TB files also helps explain the presence of AWS’s Multipart Upload service, which the company rolled out last month. Vogels explains it in more detail, as does the original blog post about it, but the gist is that users can break files in as many as 1,024 segments, ensuring that a network failure won’t kill an entire upload. Anyone who has backed up a hard drive in the cloud knows how long that upload can take; imagine a 5TB file, or a library of them.
As yesterday’s release of Android and iOS software development kits illustrates, AWS can’t afford to give up wooing developers, but this huge storage upgrade is further evidence that Amazon has its eyes set on winning bigger customers with bigger computing needs. I’ve said in the past that AWS needs a PaaS offering, but I’m beginning to question just how much it does. It doesn’t need to rely on developers if it can tap into huge amounts of infrastructure spending by high-performance users.
Image courtesy of Flickr user makelessnoise.
Related content from GigaOM Pro (sub req’d):