Reducing AWS Fargate Startup Times with zstd Compressed Container Images

Updated Oct. 19, 2022: Amazon ECR’s Enhanced Scanning, powered by AWS Inspector, now supports scanning zstd compressed container images

AWS Fargate is a serverless compute engine for containerized workloads running on Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS). Once a containerized workload has been scheduled by a container orchestrator, AWS Fargate provides a serverless compute environment to run that workload removing the operational overhead to secure, scale, and manage Amazon EC2 instances.

On AWS Fargate, each containerized workload, Amazon ECS Task, or Kubernetes Pod, runs on its own single-use single-tenant instance that’s not reused after the workload finishes. The container images required to run a workload on AWS Fargate are downloaded for every Amazon ECS Task or Kubernetes Pod. This process is in contrast to multi-tenant instances like Amazon ECS Container Instances or Kubernetes Nodes, where a container image may already exist on a host from a replica of the same workload. The requirement to download and extract an image each time can lead to workload startup times being slower on AWS Fargate when compared to a multi-tenant instance. It’s in this area of extracting container images where zstd-compressed container images help reduce start up times on AWS Fargate.

Container images are distributed as a bundle of layers that are stacked on top of each other as an overlay filesystem at runtime. A container image builder compresses each image layer before pushing it up to an image registry, such as Amazon Elastic Container Registry (Amazon ECR). By default, container image builders compress each container image layer using the gzip compression algorithm. When the container runtime downloads each image layer, the runtime decompresses the contents into the container runtime storage system.

Container image builders and container runtimes support an alternative compression algorithm for image layers, zstd. zstd is a compression algorithm developed by Meta, whose own benchmarks show that zstd can achieve higher compression ratios and higher decompression speeds than the gzip compression algorithm.

In our internal testing of zstd compressed container images on AWS Fargate, we have seen up to a 27% reduction in Amazon ECS Task or Kubernetes Pod startup times. The reduction in startup time varies by container image, with the larger container images seeing the greatest improvement. Uses cases like machine learning, artificial intelligence, and data analytics traditionally have large container images. Consequently, these workloads could see the most benefit from adopting zstd compression.

Building container images using the zstd compression algorithm

There are multiple tools to build container images, such as Docker Build, buildkit, and Buildah. In this post, we use buildkit to build zstd compressed container images. Work to support zstd container images in the Docker Engine’s native image builder, exposed by docker build, are ongoing and can be followed here.

Buildkit is a standalone container image builder that can run on a Linux machine or in a containerized environment. The most convenient way to interact with buildkit is through a Docker cli plugin called Docker Buildx. Docker Buildx is preinstalled on any Docker Desktop installation, and can be manually installed on a Linux host running the Docker Engine. Once Docker Buildx has been installed, the commands to build a container image with zstd compression are very similar to those familiar with the docker build tool set.

# First create a Buildkit Container Image Builder
$ docker buildx create \
  --name zstd-builder \
  --driver docker-container \
  --driver-opt image=moby/buildkit:v0.10.3
 
# Switch the Buildx Context to the new Builder
$ docker buildx use zstd-builder

# Then assuming you are within your application source code directory
$ docker buildx build \
  --file Dockerfile \
  --output type=image,name=$IMAGE_URI:$IMAGE_TAG,oci-mediatypes=true,compression=zstd,compression-level=3,force-compression=true,push=true .

Breaking apart the --output flag in the command above:

• type=image – Instructs buildkit to produce a container image
• name=$IMAGE_URI:$IMAGE_TAG – The image name and image tag of the finished container image
• oci-mediatypes=true – The container image will be built to the OCI v1 image standard (instead of the default Docker Image v2 Schema 2 standard)
• compression=zstd – Use the zstd compression algorithm instead of the default gzip compression algorithm.
• force-compression=true – This flag is required to force the container image builder to recompress image layers that have previously been stored in the registry as gzipped archives, for example container base images.
• compression-level=3 – zstd has 22 levels of compression. The higher the level of compression the smaller the container image size, but comes at the cost of more CPU resources to decompress the image layer. For our use case of reducing the AWS Fargate startup times, where we need to both download and decompress the image layer before the workload can start, the highest level of compression may not produce the quickest AWS Fargate start time. It’s worth experimenting with your container images to find the optimum compression level, in our testing this was compression level 3.
• push=true – After the container image has been built, push the image to the registry. This flag is mandatory as the compression of an image happens when an image is pushed to the registry.

You can verify the compression algorithm of a container image in a registry by retrieving the image manifest with the docker buildx imagetools inspect command. Notice the mediaType of each image layer in the output below.

$ docker buildx imagetools \
    inspect --raw \
    $IMAGE_URI:$IMAGE_TAG
{
   "mediaType": "application/vnd.oci.image.manifest.v1+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:12fe1c9ed0facc73b17a4ceb431c3d160182c67f333e38d0a5677dfa1c18c4b6",
      "size": 14833
   },
   "layers": [
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:f2b553fe6ae26832d4beafd0c4ef3c4690b88f340360aef0f79800eb99ba8754",
         "size": 24229715
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:7ec3184be813e46f2cd0fa7044d232a09cca5a1594c6e3efc4c7998ad63a1cff",
         "size": 34621023
      },
      ...

Running zstd compressed container images on AWS Fargate

The AWS Fargate platform versions are used to refer to a specific runtime environment for AWS Fargate infrastructure, it is a combination of the kernel and container runtime versions. As of AWS Fargate platform version 1.4 containerd is used as the container runtime. containerd is the only dependency for running zstd-compressed container images. There are no platform or runtime configuration changes required to use zstd container images in Amazon ECS or Amazon EKS with AWS Fargate. The only workload definition change required is to update the container image referenced in the Task Definition or Pod Spec to a newly built container image that has been built with zstd compression.

Amazon EKS customers using managed or self-managed node groups can also use zstd-compressed container images as long as containerd is enabled as the runtime.

zstd container image adoption

While support for zstd layer encryption has been merged in to the upstream Open Container Initiative (OCI) Image Specification, it’s yet to make it to a release milestone. The work to formalize zstd as part of the OCI Image Specification can be found here.

Secondly, as part of a developer’s workflow not all container tools support zstd compressed container images. As the benefits of zstd compressed container images become more widely known, we expect the gaps in developer tooling to close. Popular tools that do not yet have zstd support include:

• Docker Engine – zstd-compressed container images can’t be started by the Docker Engine. The Docker Engine’s progress towards supporting zstd-compressed container images be tracked here.
• At the time of writing container image vulnerability scanners, such as Aquasec’s Trivy, Quay’s Clair, or AWS Inspector, do not support scanning zstd compressed container image layers.

Conclusion

In this post, we showed you how to use zstd-compressed container images to reduce the time taken to start containerized workloads running on AWS Fargate. With the existing support in OCI registries, container image builders and container runtimes customers do not have to change their infrastructure to leverage zstd compression container images.

• For more information on using Docker Buildx to build container Images see the Buildx documentation.
• For more information on AWS Fargate platform version 1.4 see the documentation and the under the hood blog.