Media movement for content production workloads

Authored by Greg Holick, Vice President of Product at Cloudsoda.io. The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

The media and entertainment content production process is highly orchestrated. It encompasses establishing the creative vision, scouting shoot locations, hiring crew, and procuring the right cameras and lenses to capture the action. In my opinion, the fun begins when the cameras stop rolling. That’s when terabytes of original camera files need to get from the set and into the hands of editors and Visual Effect (VFX) artists. These are the people responsible for creating the polished content we see in theaters, on streaming services, and watch on our favorite viewing devices.

Creating content during a global pandemic made us realize that workflows and processes used for years needed to change. During the Hollywood Professional Association (HPA) Tech Retreat in 2021, AWS took part in a thought leadership project called “Found Lederhosen.” The project intended to innovate and test filmmaking and creative processes in the face of the pandemic. It included cinematic shorts from London, Dubai, Brisbane, and Mexico City as well as a virtual production shoot in Hollywood, an animation sequence out of Mongolia, and a thematic video game. To make this project successful, raw content needed to get into the cloud and on the right type of storage at the right time. This made it easier for creatives to access the material from their cloud-hosted editing software. Having a solution that can move media to the appropriate storage type, and ensuring that it arrives, is crucial to supporting this workflow.

Cloudsoda.io

Cloudsoda.io (SoDA) is a data management software solution by Amazon Web Services (AWS) Partner Integrated Media Technologies that works with on-premises, hybrid, and multi-cloud environments. SoDA supports network attached storage (NAS), storage area network (SAN), object storage, and any storage mountable on Windows, Linux, and Mac operating systems for seamless archive or data migration to the cloud. SoDA can write files directly into Amazon Simple Storage Solution (Amazon S3). Customers can select their desired storage class through the SoDA user interface (UI). Storage classes include Amazon S3 Infrequent Access, Amazon S3 Glacier Instant Retrieval, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive.

SoDA AWS Architecture

The SoDA conductor can run in AWS on an Amazon Elastic Compute Cloud (Amazon EC2) instance, and deploy in a customer’s AWS account or as a software-as-a-service (SaaS) offering. Users can download agents for Windows, Linux, and macOS either on-premises or in the cloud. The SoDA conductor uses the agents to scan and move data from point to point, or from storage type to storage type.

A SoDA feature called Dry Run indexes all the files you plan to move/copy/sync from a selected location and provides an estimated cost to store and time to transfer that amount of data in the chosen Amazon S3 storage class.

The Dry Run feature shows expected storage costs and transfer times based on data volumes

Amazon S3 storage classes provide flexibility for multiple media and entertainment workflows, giving customers the ability to select the storage class that best fits their business objectives. With this flexibility, SoDA has moved tens of petabytes of content to the cloud.

SoDA for content production
As previously stated, to make a production successful, it’s crucial to get media into the cloud quickly, securely, and cost-effectively to support editors and VFX artists scattered around the globe. Earlier this month, SoDA added support for Amazon FSx for Windows File Server and Amazon FSx for Lustre to help solve a pain point identified during HPA 2021.

We began working with the Amazon Nimble Studio (Nimble Studio) team to create a way for customers to quickly get content into a VFX pipeline. SoDA provides an efficient way to move data from on-premises storage or a Digital Imaging Technician’s (DIT) on-set cart to the Amazon FSx for Lustre file system, which provides fully managed shared storage.

FSx for Lustre can link to an Amazon S3 bucket and present the objects in the bucket as files using AWS’ extended metadata in the objects. When moving data to Amazon S3, SoDA has implemented these metadata tags so users can see the file metadata in FSx for Lustre and hydrate the media from an Amazon S3 bucket. SoDA uses multipart upload and dynamic threading to optimize transfer of media assets to the cloud. SoDA overcomes the issues of latency and network challenges users face from home and makes it easy to upload media to Amazon S3. SoDA is enhancing its User Datagram Protocol (UDP) capabilities for agent-to-agent transfers, which will allow movement of media from on-premises directly to an FSx file system.

SoDA can replicate data between AWS Regions, allowing VFX Artists to work on content from any location. When VFX projects in Nimble Studio are complete, SoDA can archive data to Amazon S3 Glacier storage tiers to save on costs to customers while protecting their assets. If a project is paused in the middle of production, SoDA can move data to Amazon S3 Glacier Instant Retrieval from the active FSx for Lustre file system, and back when production resumes.

Customer Benefits
With the integration of Amazon FSx file systems, including FSx for Lustre, into SoDA, it’s easier to get media on the right type of storage for editors and VFX artists and accelerates content production workflows on a global scale. SoDA is available on AWS Marketplace. Learn more about SoDA on AWS.