GigaOm Use Case Scenario for Decentralized Object Storage for Video

Best Practices and a Practical Example for Implementation

Table of Contents

  1. Summary
  2. Report Methodology
  3. Design Criteria
  4. Additional Considerations
  5. Solution Profile
  6. Analyst’s Take

1. Summary

In the media and entertainment (M&E) industry, video productions invest heavily in large-scale infrastructures to store vast amounts of data, both in the cloud and on-premises. Video and other media assets coming from different sources have to be edited and rendered into the final product by teams that are often globally distributed. Depending on how the videos will be consumed, the final product is rendered in multiple versions and kept for long periods of time afterward. This challenge is even bigger now with videos that are shot at 4K and 8K resolutions.

No matter the size of the company, media-rich content requires a lot of storage capacity. It has to be reliable, fast, and, at the same time, reasonably priced. Object storage is considered one of the best options for storing unstructured data due to its scalability, cost, simplicity, and accessibility, but it also poses challenges, especially when data needs to be globally accessed and distributed. This GigaOm Use Case Scenario report explores the application of decentralized object storage in video production and collaboration.

About the GigaOm Use Case Scenario Report

This GigaOm Use Case Scenario report focuses on specific scenarios and best practices to improve adoption of technologies, exploring both use case design criteria and a viable technical solution. In this context, a particularly demanding use case for decentralized object storage can be found in video production and collaboration.

Use of video is growing in every industry for a variety of reasons: surveillance, training, marketing, conference calls archiving, and so on. These are generic use cases found in organizations of every size, but when we focus on the M&E industry, and specifically on video production, we find that:

  • Videos are recorded and edited in different locations.
  • Users take advantage of compute resources from different providers to render videos that must be centralized in a single location.
  • All video archives are now nearline.
  • Content has to be distributed efficiently on different channels and platforms.
  • Users don’t use S3 protocol directly, but want familiar file interfaces (SMB or NFS) and media asset management tools (MAM) to simplify their workflows.

These requirements create a formidable problem. Meeting high-level standards in terms of availability and resiliency can be challenging, especially when the total cost of the infrastructure must be accounted for. In this regard, the user should take into account some important aspects:

  • Infrastructure resiliency: Object storage is usually resilient, but it is crucial to consider business continuity and disaster recovery for on-premises/hybrid infrastructures.
  • Data accessibility and availability: Even in the public cloud, having multiple copies of data can be a requirement in case of a zone or region failure. Keeping data synchronized is very expensive and creates additional synchronization issues if data needs to be accessed concurrently from multiple locations or for backup reasons.
  • Performance: Even though performance is not usually associated with object storage, parallelism and throughput are important characteristics to consider, especially when video is involved. Content delivery networks (CDN) are a solution, but they are expensive and complicate the infrastructure topology.
  • Scalability: This can be an issue for on-premises deployments, especially for large systems installed in locations with limited space.
  • Cost and TCO: Cost can be one of the biggest issues when video is involved, especially in hybrid and public cloud environments because of complex billing mechanisms and egress fees.

Decentralized storage is a solution to these challenges. A decentralized storage system is based on a peer-to-peer (P2P) network, a type of architecture that has found some success for data distribution and file sharing. Instead of storing data in a centralized system made up of data centers, it is chunked, distributed, and stored on thousands of nodes in a global network or the internet. Figure 1 compares traditional shared storage with decentralized storage. This latest version of decentralized cloud storage has evolved greatly and is now enterprise-grade and considerably more secure, performant, private, and durable than a centralized cloud provider. It is also a fraction of the cost.

Figure 1. Traditional and Decentralized Storage

We have seen several attempts over the past decade—largely unsuccessful—to build a decentralized, or P2P, network infrastructure. But risks are generally mitigated by the large number of unused commodity resources across the internet, better security, and blockchain technology that ensure data immutability and consistency. It is now easier to take advantage of the abundance of unused, and sometimes unreliable, resources to build performant and secure storage infrastructures. More so, the interest in web3 and decentralized internet technologies has attracted large investments, accelerating product development and the growth of a solution ecosystem.