We build DCC integrations for render pipelines. Here’s how easy it was with AWS Deadline Cloud

This blog was authored by Gunpowder, LLC.

Creating an integration for your DCC with AWS Deadline Cloud? One word: templates.

Introduction

In the world of creative media and visualization, rendering is a crucial step that brings artistic visions to life. However, the process of integrating various creative tools and renderers into a unified rendering pipeline can be a complex and time-consuming endeavor.

This complexity stems from the diverse suite of digital content creation (DCC) tools employed by visual effects studios and animation houses, each producing distinct visual styles. Even separate teams within the same company often tackle unique projects with different tools. Keeping pace with the latest technological advancements and industry standards is essential; failing to do so can quickly result in becoming entrenched in outdated workflows. Without flexibility to adopt the tools they want, studios face stymied creativity and operational inefficiency, ultimately impeding their ability to deliver the highest quality visual content.

Gunpowder LLC specializes in providing infrastructure and pipeline solutions to power ambitious creative workloads, and recently had the opportunity to collaborate with Amazon Web Services (AWS) on its new render management service, AWS Deadline Cloud. The team relied upon its extensive production experience in rendering workflows to use and help build upon the Open Job Description (OpenJD) specification and the Deadline Cloud developer tools.

Overview

The AWS team developed OpenJD, announced in February 2024, as a flexible and open specification for defining render workflows and building related developer libraries. The goal of OpenJD is to simplify the integration of creative tools and renderers into rendering pipelines. As a trusted partner in the creative industry, Gunpowder was invited to help implement a “starter set” of v1 render submitters for pipeline developers to adapt for a wide range of tools and renderers, including SideFX Houdini, Yeti, Maxon Cinema4D, KeyShot, Rhino, Renderman, Autodesk Arnold, Karma, Mantra, Chaos V-ray, Cycles, Octane, Redshift, and Corona.

The solution

The AWS Deadline Cloud developer tools working with the OpenJD specification provide abstractions for common patterns in distributed rendering. These abstractions take the form of reusable templates, allowing for streamlined development and reduced coupling between the render service API and the creative tool APIs.

One of the key advantages of OpenJD is its ability to model render dependencies, execution environments, and smart render adaptors that manage the state of a render job on worker nodes. This functionality—typically opaque and idiosyncratic in traditional render farm products—is generalized and made accessible through the OpenJD templates.

Gunpowder’s deep expertise in creative tool APIs allowed the team to navigate these interfaces efficiently and implement solutions to test on the Deadline Cloud service. The process involved writing common render workflow patterns in template.yml files and using the creative tool APIs to query the scene and populate templates with specific values.

For example, a common render workflow in SideFX Houdini is to export geometry prior to rendering an image sequence. In Houdini, this is modeled using ROP (Render Output) nodes with a geo node connected as input to the render node. We use the Houdini object model to parse the ROP network and translate this relationship into an OpenJD template step dependencies list:

- name: mantra1
  dependencies :
   - dependsOn: geo1

Submitter logic also collects scene assets and includes them in the payload and job template sent to the rendering service.

Each DCC has its own unique API accessing asset references in the scene description. With Houdini, we use the hou.fileReferences() method to collect and then serialize required assets to the OpenJD bundle asset_references.yaml file.

Challenges and tasks

Because Gunpowder needed to provide integrations for diverse tools/renderers and optimize the submission process for each, the team tackled several challenges in the process to solve for the various issues creative studios face.

●  One of the primary tasks was to ensure seamless integration between diverse creative tools and renderers, each with its own unique set of APIs and requirements. The team accomplished this by leveraging our domain expertise in creative applications to author consistent python modules per DCC for accessing common scene information such as file references and renderer parameters.

●  In addition, the project required support for creative tools that host a diverse set of render products without duplicating logic. Gunpowder achieved this by authoring job templates that capture the broadest use cases. For example, a single Maxon Cinema 4D render template supports at least eight different render engines, including Autodesk Arnold, Redshift, Chaos V-ray, Corona, Octane and the native Cinema 4D rendering engine. Your experience with each combination will vary based on your own individual pipeline.

●  Another challenge was handling the variability in processor architecture, platform, and plugin support. Creative tools, render plugins, and render engines all vary in their support across these dimensions, so care was taken to ensure submitters and adaptors produced the expected output.

Testing the submitter logic and job templates used a novel approach, calling on the Custom Fleets feature of AWS Deadline Cloud. This allowed the team to commission its own render worker nodes. Each custom fleet supported a different class of processor architecture, operating system and creative tools suite. For example, a gpu-linux fleet tested Redshift renders, while a cpu-windows fleet tested Keyshot renders.

Result

Working with the OpenJD specification and the Deadline Cloud developer tools allowed Gunpowder to build a set of starter render-submitter examples that customers can download, use to experiment, and explore customization. The adoption of the template system streamlined the development process, minimizing the variability in development times that is often encountered by production studios when evolving their pipelines under tight deadlines characteristic of production projects. This method ensures a more efficient and predictable workflow, crucial for maintaining momentum and meeting project milestones with reliability. What’s more, it allows us (and anyone who downloads them) to continue evolving and adapting future versions of these integrations as more customers adopt and test them against their own complex pipelines.

Conclusion

The integration of the Open Job Description standard into creative rendering pipelines represents a significant step forward in simplifying and standardizing the development process. By decoupling render job submissions from specific render farm solutions, OpenJD empowers studios and developers to build integrations once and deploy them across multiple rendering pipelines.

“At Gunpowder, we are excited to have played a role in this groundbreaking project and to contribute our expertise to the development of the OpenJD specification,” said Tom Taylor, Founder of Gunpowder LLC. “We encourage the creative community to embrace and explore the potential of this open standard, as it promises to unlock new levels of flexibility, interoperability, and efficiency in the rendering and visualization workflows of the future.”

Ready to experiment?

You can find the set of starter plugins to create jobs using each plugin’s corresponding software. Note: every pipeline is different, and sometimes, different versions of the same software have different requirements in a pipeline, so please test in an isolated deployment—and don’t hesitate to customize at the code level to suit your needs. We welcome your feedback at tech@gunpowder.tech.

Authored by Gunpowder, LLC. The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.