Introducing multi-architecture container images for Amazon ECR

Containers are a de facto standard in cloud application development and deployment. Publishing software in container images provides developers an integrated packaging solution, bundling software and all required dependencies into a portable image format. This image can then be run anywhere, abstracting away the infrastructure-specific aspects of deployment.

However, the promise of running anywhere only goes so far. Some applications have specific host platform or operating system requirements, such as supporting both Linux and Windows. Compute architecture is another variable, especially with the compelling price-performance ratio of AWS Graviton ARM-based instances running in EC2. Before today, such container images had to be published and deployed to Amazon ECR using architecture-specific naming conventions, complicating some aspects of the image lifecycle.

Today we are announcing multi-architecture container images for Amazon ECR. This is a much-anticipated feature that makes it simpler to deploy container images for different architectures and operating systems from the same image repository.

Container images under the hood

Amazon ECR is a fully managed container registry that makes it easy for developers to store, manage, and deploy container images. It is highly available, scalable, and simple to use. Before discussing multi-architecture images in detail, let’s first cover some underlying aspects of how container images work.

The term ‘container’ refers to a set of operating system components, computing resources, and configuration to provide an isolated compute environment for processes to run within. One of the resources specified for a container is its file system. When we refer to a container image, it’s this file system in a portable format along with container configuration and other metadata that we refer to. Popular container development tools like Docker allow developers to create container images that contain software or a service and all of the required dependencies, which is what makes containers such a portable option.

Container images consist of two main parts, layers and a manifest. Each container image has one or more layers of file system content. The manifest specifies the layers that make up the image as well as its runtime characteristics and configuration. The container image format for Docker is defined by the Docker Image Specification and the related Image Manifest Specification. The Open Containers Initiative went on to define the runtime-agnostic OCI Image Specification.

Image registries like Amazon ECR store images which adhere to these specifications in repositories, and each specific image is referenced by one or more tags. Images are typically tagged in order to specify a version of the software or service when pushing and pulling the image. Putting this all together, when you first pull a container image for use in Docker or another container runtime, two things happen. First the manifest is pulled locally based upon the specified image repository and tag, and then the manifest is used to assemble the container file system from the layers specified.

For a concrete example, you can use the docker inspect <image> command to see the manifest of any local image in your Docker development environment. As you can see, platform characteristics such as architecture and operating system are clearly specified by the image manifest. So then, how do you easily deploy containers across different operating systems and platform architectures?

Before today, when publishing images to a repository in Amazon ECR these characteristics had to be specified in the image tag. Alternatively, you could store platform-specific images built from the same source in their own image repositories. The correct version of the image for your compute environment would then need to be pulled by explicit reference, for example {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image-linux-arm64:2.7 rather than {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image:2.7. This is inconvenient, and requires OS- or architecture-specific references throughout the entire development and deployment lifecycle of your image.

Introducing multi-architecture images in Amazon ECR

With multi-architecture image support in Amazon ECR, it’s now easy for you to build different images to support multiple architectures or operating systems from the same source and refer to them all by the same abstract manifest name. This is achieved through support of an image specification component known as a manifest list, or image index.

Manifest list support has been present in the Docker Image Manifest Specification since V2 image manifest (schema version 2). It is also included in the Open Containers Initiative Image Specification v1, though it is referred to there as an image index. A manifest list (or image index) allows for the nested inclusion of other image manifests, where each included image is specified by architecture, operating system and other platform attributes. This makes it possible to refer to an image repository that includes platform-specific images by a more abstract name, for example {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image:2.7.

The container engine responsible for creating the container pulls from the registry the correct layers for the compute environment where it’s running based upon the values in the manifest list. With a small number of additional steps during your image builds, your clients can pull an image by version tag and get the correct image for the platform they are running on. This can simplify both your Amazon ECR repository management as well as your CI/CD pipelines, as mentioned above. These additional steps are detailed below.

Working with multi-architecture images in Amazon ECR

In this walkthrough, you will create two container images, one for x86_64 (64-bit x86-based systems) and one for aarch64 (64-bit ARM-based systems). You’ll then push these images to a repository in Amazon ECR and then create a manifest list referring to each by their architectures. Finally, you’ll pull an image by manifest list name without needing to specify the correct architecture.

To get started, you need:

• An AWS account and the aws CLI installed and configured for use in your development environment
• A repository in Amazon ECR named hello (or your own repo name). For information about creating a repository, see the Amazon ECR documentation.
• A Docker development environment and familiarity with using Docker
• Two EC2 instances, one x86_64 and the other aarch64. I’m using T3 and A1 instances, respectively. For more information on launching instances, see the Amazon EC2 documentation.

To make cutting and pasting the following commands easier, set the following environment variables in your shell to refer to your numeric AWS Account ID and the AWS Region where your registry endpoint is located.

$ AWS_ACCOUNT_ID=aws-account-id
$ AWS_REGION=aws-region

Now, you will need to build images for two different architectures in your Docker development environment. You can use your own, or clone this handy hello world app for the purposes of this walkthrough.

$ git clone https://github.com/jlbutler/yahw.git && cd yahw
$ make all
$ docker images hello
REPOSITORY TAG   IMAGE ID       CREATED         SIZE
hello      arm64 8d2063eddc5e   17 seconds ago  7.19MB
hello      amd64 cbfda9e83a41   27 seconds ago  7.59MB

Note that the walkthrough images use architecture-specific tags. This is for the purpose of the demonstration only, it’s best to tag your images with an explicit version or another meaningful reference.

Now that you have your images for each platform, tag them to refer to your repository in Amazon ECR. Log your Docker client into ECR as needed.

$ aws ecr get-login-password --region ${AWS_REGION} | \ 
    docker login --username AWS --password-stdin \ 
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello 
  Login Succeeded

$ for i in amd64 arm64; do
for> docker tag hello:${i} ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:${i}
for> done

With the images tagged with your repository path, they are ready to push to Amazon ECR.

$ docker images | grep hello
REPOSITORY                                                    TAG    IMAGE ID      CREATED        SIZE
{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello     arm64  8d2063eddc5e  4 minutes ago  7.19MB
hello                                                         arm64  8d2063eddc5e  4 minutes ago  7.19MB
{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello     amd64  cbfda9e83a41  4 minutes ago  7.59MB
hello                                                         amd64  cbfda9e83a41  4 minutes ago  7.59MB

$ for i in amd64 arm64; do
for> docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:${i}
for> done

You can verify that your images pushed with the aws ecr command.

$ aws ecr --region ${AWS_REGION} describe-images --repository-name hello
{
    "imageDetails": [
        {
            "registryId": "aws-account-id",
            "repositoryName": "hello",
            "imageDigest":"sha256:b50bd7f7..5a0dc770",
            "imageTags": [
                "amd64"
            ],
            "imageSizeInBytes": 3954211,
            "imagePushedAt": "2020-04-24T17:24:11-04:00"
        },
        {
            "registryId": "aws-account-id",
            "repositoryName": "hello",
            "imageDigest": "sha256:2f333a8b..27fc2172",
            "imageTags": [
                "arm64"
            ],
            "imageSizeInBytes": 3702268,
            "imagePushedAt": "2020-04-24T17:24:25-04:00"
        }
    ]
}

At this point, you could pull these images by their architecture-specific tags, but let’s simplify by creating a manifest list and pushing it to Amazon ECR.

In your Docker development environment, create a new manifest list for this image set with the docker manifest create command. If you do not have experimental features enabled in your client, you need to do this first. For more information, see the Docker documentation.

$ docker manifest create ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello \
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:amd64  \
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:arm64 
Created manifest list {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest

The manifest is using the default latest tag here, but again you should use something that more explicitly references this version of the container for production purposes.

One little bit of housekeeping we need to do is to annotate the manifest so that the manifest list correctly identifies which image is for which architecture. An artifact of the Docker build command is that it sets the architecture to the build environment’s architecture, even if cross-compiling for another architecture as is the case in our example. But, there is a simple remedy for this via the docker manifest annotate command.

$ docker manifest annotate --arch arm64 ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello \
      ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:arm64

Before you push it, you can inspect your newly-minted manifest and note that it has a manifests list with two distinct image references, each with digests mapping to your images and with their appropriate platform.architecture values.

$ docker manifest inspect ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 528,
            "digest": "sha256:b50bd7f7..5a0dc770",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 528,
            "digest": "sha256:2f333a8b..27fc2172",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

After confirming your manifest is ready for use, push it to your repository in Amazon ECR as you would any image.

$ docker manifest push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello
sha256:bd7a61a6ea3c366c0e58d70233900f0c761d6051da0ad8cbaa19011f873c37bc

With that final step, your images are ready to pull by referring to the higher-level manifest image tag. Head over to your Graviton ARM-based EC2 instance, log your Docker client into Amazon ECR if needed, and pull the image by its latest tag. Note you’ll need to set your AWS_ACCOUNT_ID and AWS_REGION accordingly.

$ docker run -d -p 8080:8080 ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:latest
Unable to find image '{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest' locally
latest: Pulling from hello
2f12463f1928: Pull complete
Digest: sha256:bd7a61a6ea3c366c0e58d70233900f0c761d6051da0ad8cbaa19011f873c37bc
Status: Downloaded newer image for {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest
fd4c0ae177dad82315cbcc7f6acb8fdcb733318c7f2e1bb832693de896e3527a

$ curl localhost:8080/hello
{"arch":"aarch64","message":"Hello, there!","os":"Linux 4.14.173-137.229.amzn2.aarch64"}

Next steps

From the above walkthrough, you can see that adding a few simple steps to your container build pipelines means that you no longer need to use OS- and architecture-specific image repositories or tags to ensure that your containers are deployed to their correct runtime environment. This simplifies your container pipelines considerably, enabling simpler image naming conventions for your published container images.

When working with interpreted languages such as Python, it is likely that your image builds for different platforms are fairly transparent, simply referencing the correct base images in your Dockerfiles. Keep in mind there may be some caveats, depending upon your use of system-level primitives which may be OS-specific.

For container builds with compiled languages such as C++, there are OS- or architecture-specific build steps. Some languages provide cross-compiling features that make this process simpler, as in the example’s use of Go. This allows you to build for various operating systems and architectures all from one system. If your development environment is Docker Desktop, you may also consider the docker buildx command, which simplifies multi-architecture builds by using QEMU’s emulation features during build.

Regardless of programming language, you can pass any required platform details into a build using the —build-arg option of the docker build command combined with the Dockerfile ARG directive. Or, you can use platform-specific Dockerfiles and explicitly reference them via the —file option of docker build. For more information on using these options, see the Docker build documentation.

As mentioned above, images should be tagged based upon their version or another specific identifier, such as git commit ID. With multi-architecture image support, your builds and deployments can refer to a single image name and a version-specific tag, no longer needing to refer to operating system, architecture, or other platform details.

Let us know what you think of this new feature, and check out more upcoming features on the AWS containers roadmap.