Diving into OCI Image and Distribution 1.1 Support in Amazon ECR

AWS recently announced that Amazon Elastic Container Registry (Amazon ECR) now supports version 1.1 of the Open Container Initiative (OCI) Image and Distribution specifications. This latest version includes support for image referrers, as well as significant enhancements for distribution of non-image artifacts.

We are excited about this set of new capabilities, which helps customers more easily manage content related to their container images. With these updates, customers can push image signatures, software bill of materials (SBOMs), attestations, and other content related to a specific image right alongside their images in Amazon ECR. Customers can then manage these artifacts with core Amazon ECR features, and easily look up artifacts by image reference and pull them from Amazon ECR to any build, test, or workload environment.

Basing these new capabilities on the OCI 1.1 specifications means that a wide array of OCI-compliant open source tools work seamlessly with Amazon ECR. In this post we’ll offer a brief overview of how Amazon ECR implements support for the Open Container Initiative standards. Then we’ll dive into some technical details and a specific use case that will illustrate how you can get started with these new features right away.

Open Container Initiative and Amazon ECR

Bringing these features to Amazon ECR started more than two years ago. AWS employees collaborated upstream within the Open Container Initiative (OCI) community on what has now become the first feature update of both Image and Distribution specifications. These specifications exist to standardize containers so that services, tools, and code are portable and can interoperate dependably.

Today, OCI curates and maintains three main specifications, Runtime, Image, and Distribution. These new Amazon ECR features relate to the latter two; where Runtime is concerned with how containers are launched and run on a host, Image specifies how container images are defined, and Distribution specifies how images are pushed and stored in registries as well as how they are pulled to create workloads in your build and compute environments.

Most clients connect to Amazon ECR using an OCI-compliant tool like docker or finch, or a container runtime like containerd. With these tools, when you push and pull images you are interacting with Amazon ECR’s OCI- and Docker-compatible endpoint. This endpoint implements the Open Container Initiative Distribution specification APIs, as well as the Docker Registry HTTP API v2 which the OCI standard is based on. All of this enables open source tools to implement against a standard set of interfaces, instead of requiring cloud-specific SDKs or other provider-specific integrations.

Introduction of Artifacts in Amazon ECR

Amazon ECR is continually evolving to support non-image artifact use cases as they gain popularity. In 2020, we added support for OCI artifacts, which introduced non-image content distribution for artifacts like Helm charts in Amazon ECR. Tools like Helm leverage this support by taking advantage of the flexibility of the OCI Image specification to store the artifact type in the manifest, specifically in the configuration section. This metadata in an image manifest is used to describe the container configuration for the image that is stored. In the case of a Helm chart or image signature, no container configuration is required. This provides an easily overloadable piece of metadata that clients and registries can use, specifically this is the config.mediaType field. Tools can use this field to specify to a registry or other clients the type of the artifact content, for example a Helm chart. Here is an example manifest of a Helm chart. Note the config.mediaType is set by the Helm client to indicate that this image is in fact a Helm chart.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:02a274bb7486be1dc856f81229fea274fa0e8f92e84dbf6c5291bb512b7944b5",
    "size": 233
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
      "digest": "sha256:62ab5a7a28fbfb9ae79bf3205375f07fab25261d8de83862fad381c3a051c1e4",
      "size": 38507
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-07-10T16:15:09-07:00",
    "org.opencontainers.image.description": "A New OCI Service Chart",
    "org.opencontainers.image.title": "oci-service-chart",
    "org.opencontainers.image.version": "1.0.0"
  }
}

While this trick in OCI Image 1.0 has worked well, it’s not obvious to all client developers and it was not consistently implemented. With OCI Image 1.1, we now have a more obvious and stable artifactType field stored directly on the Image manifest. This is a single point of reference in the manifest that all clients and registries can use to store and look up the type of content. Amazon ECR will continue to support use of the config.mediaType field for clients that rely upon it, but we feel that client tooling will broadly adopt this new enhancement.

Working with Referrers in Amazon ECR

With the advent of supply chain security solutions for containers and the need for standardizing registry support for solutions like container image verification and SBOM publication, managing the reference relationships between artifacts and images became a requirement. In addition to enhancing support for non-image content, the work in the Image and Distribution specifications also defined a method for clients to associate non-image content with images in a given registry.

To provide for this, OCI Image 1.1 has another new manifest field, subject, which provides a way to persistently store the digest of a referred-to image within an artifact’s manifest. Now a client can specify both content type, and optionally an image that the content refers to. As a peer feature enhancement, OCI Distribution 1.1 introduced a new referrers API endpoint for registries to implement. With this endpoint, clients can query a registry like Amazon ECR for any image referrers for a given image digest. This enables clients to ask the registry if an image has any referrers, walk a list of referrers by type, and to retrieve any referring artifacts of interest. For example, the notation image signing client from the Notary project implements a notation verify command which queries for any signatures related to the image specified, downloads any discovered signatures, and verifies the image content all in a single command.

The way Amazon ECR treats reference artifacts is mostly the same as an image. Customers push and pull all content using the same APIs, and referrer artifacts are returned by aws ecr describe-images, and they will show up in the console just as other content. Amazon ECR’s replication feature replicates referrers to configured destinations on push so that image signatures, SBOMs, and other referrers will be present in any repository where you replicate images across accounts or regions. To help with the lifecycle management of an image’s reference artifacts, Amazon ECR Lifecycle Policies (LCP) automatically clean up artifacts within 24 hours of a subject image deletion. Additionally, reference artifacts that refer to an active image are protected from deletion by LCP rules until their subject image is deleted.

To differentiate between referrers and images when they are pushed to a repository, Amazon ECR now emits a new detail-type in EventBridge events for pushes and deletions of reference artifacts. This allows EventBridge rules to easily target images for deployment-related actions, or for specific types of reference artifacts to be used in build or deployment workflows.

An example event that would be emitted after pushing a reference artifact is shown here.

{
  "version": "0",
  "id": "25fcd817-5f92-89f2-e2dc-47f5f407503d",
  "detail-type": "ECR Referrer Action",
  "source": "aws.ecr",
  "account": "239609618315",
  "time": "2024-06-27T17:52:58Z",
  "region": "us-west-2",
  "resources": [],
  "detail": {
    "result": "SUCCESS",
    "repository-name": "oci-helloworld",
    "image-digest": "sha256:8f39804aab5a4814a97b674e55b5aa0046f86de069d230ba80b86c088eb2503a",
    "action-type": "PUSH",
    "artifact-media-type": "application/vnd.cncf.notary.signature",
    "image-tag": "",
    "manifest-media-type": "application/vnd.oci.image.manifest.v1+json"
  }
}

Referrers by example with Amazon ECR and AWS Signer

To help make more sense of these details, let’s look at a working example with image signing. AWS Signer is a fully managed code signing service to help you ensure the trust and integrity of your code. Customers can sign their content, which involves the creation and storage of a digital cryptographically verifiable signature. This signature can then later be used to verify the content to ensure that it is unaltered and from a trusted publisher.

Last year, AWS Signer introduced support for container image signing, integrating their signing service with the Cloud Native Computing Foundation’s (CNCF) Notary project’s open source Notation client. This client uses the OCI specification to interact with registries to store, look up, and retrieve image signatures. Until now, this solution leveraged the OCI 1.0 trick described previously, using the manifests config.mediaType. It also used a bit of a trick in how it stored and looked up signatures, leveraging an OCI Image Index (or Docker Manifest List), which is typically used to store images for multiple operating systems and architectures for a given image version. Notation supports OCI 1.0 registries by overloading that index to store referrers in a dedicated index. This all becomes much simpler for customers with the new OCI 1.1 features in Amazon ECR, as we’ll show soon.

Pushing a signature to Amazon ECR as a reference artifact works similarly to pushing any other content. While this is typically a simple Command Line Interface (CLI) invocation for customers, e.g. finch push, under the covers Amazon ECR’s OCI endpoints are used to first store content in one or more layers, or blobs, and then to upload a manifest which describes that layered content. It’s important to note that both the blobs and manifests are immutable, and are verifiable via SHA256 checksum. The manifest’s checksum is what we are referring to when we call out a digest.

It’s useful to understand a bit of the context between clients and registries. In the example of signing an image with AWS Signer, a specific image is being signed. The Notation client will retrieve that image’s manifest and create an image signature based on its verifiable and immutable content. The signature is then uploaded to the registry in two steps. First, the signature itself is encoded as a layer, or blob, and uploaded by the client to the registry. Then the client creates a manifest that describes the immutable content of the signature, and uploads that to the registry as well.

Let’s get started: Signing and verifying images with AWS Signer

All of the steps above are typically abstracted for you by the client implementation. So you can see first-hand how this works, let’s walk through a simple demonstration of signing a container image in Amazon ECR with AWS Signer. As you will see, once the tools are installed and configured, it’s a single CLI invocation to sign or verify a container image (e.g. notation sign, notation verify). We’ll make use of another open source tool, oras, to peek under the covers and see how things are working.

To get started, first make sure that your AWS CLI is updated to the latest version. Instructions are found at Installing or updating the latest version of the AWS CLI in the AWS Command Line Interface User Guide.

Next, install the Notation CLI and the required AWS Signer plugin and root certificate. The AWS Signer Developer Guide provides a list of installers for Linux, macOS, and Windows. For the example below, the macOS arm64 version was used.

Finally, to see some of the details and work directly with content in Amazon ECR, install the CNCF ORAS project’s oras client. Installation instructions can be found in the project documentation. Note that oras is not required for signing and verifying images with AWS Signer. We’re using it here to reveal some of the underlying details and it is entirely optional.

At the time of writing, the following client versions were used. Note that notation support for the Referrers API is only available in 1.2.0, currently available as a pre-release on GitHub.

$ aws --version
aws-cli/2.17.6 Python/3.11.8 Darwin/23.5.0 exe/x86_64

$ notation version
Notation - a tool to sign and verify artifacts.

Version:     1.2.0-alpha.1
Go version:  go1.22.4
Git commit:  2f4387276b4a73fb4b81f9499afe0aa156b56218

$ oras version
Version:    1.2.0+Homebrew
Go version: go1.22.3

Now is a good time to verify the AWS Signer plugin is installed and configured in notation. If you used one of the curated installers, this will have been done automatically as part of installation.

$ notation plugin ls
NAME                                   DESCRIPTION                      VERSION   CAPABILITIES                                                                                             ERROR
com.amazonaws.signer.notation.plugin   AWS Signer plugin for Notation   1.0.298   [SIGNATURE_GENERATOR.ENVELOPE SIGNATURE_VERIFIER.TRUSTED_IDENTITY SIGNATURE_VERIFIER.REVOCATION_CHECK]   <nil>

Finally, before you can sign images with AWS Signer you’ll need to create a Signing Profile using the Notation platform. This is a unique resource that you use to perform signing jobs. You only need to create a signing profile once, and it can be revoked and canceled at any time.

$ aws signer put-signing-profile --profile-name demo_signer --platform-id Notation-OCI-SHA384-ECDSA

With these tools installed and configured, you’re ready to start signing and verifying images in Amazon ECR.

First, log into Amazon ECR with your notation client. You can log your oras client in now as well.

REGISTRY_ID=<Your AWS account ID>
AWS_REGION=<Your preferred AWS region>

$ aws ecr get-login-password --region ${AWS_REGION} | notation login -u AWS --password-stdin ${REGISTRY_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
Login Succeeded
$ aws ecr get-login-password --region ${AWS_REGION} | oras login -u AWS --password-stdin ${REGISTRY_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
Login Succeeded

When you use notation to sign an image in Amazon ECR with AWS Signer, the client is signing the immutable manifest for the image. Since image tags (e.g. latest) move around, it’s best to sign using the verifiable and unique image digest of that manifest. This can be viewed in the console or retrieved with the AWS CLI.

Choose an image to test signing with, or push a new image. You can then retrieve the digest with the AWS CLI using the following command.

$ aws ecr describe-images --repository-name oci-helloworld --image-ids imageTag=1.0 --query 'imageDetails[*].imageDigest'
[
    "sha256:54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d"
]

You can also easily calculate the digest with the shasum utility using the content of the image manifest as shown here.

$ oras manifest fetch 12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld:1.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 973,
      "digest": "sha256:ad9641dff0fd658756bacbbe2379ad927ccb6ef7eb394db5ecc896fd5827318c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3473328,
         "digest": "sha256:44343b498b662ec241ddcc5461f935f5e0030a2b552ba6f83df696ee9e2d9956"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 32,
         "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
      }
   ]
}
$ oras manifest fetch 12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld:1.0 | shasum -a 256
54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d  -

Once you have the image digest, you can use notation with the AWS Signer plugin and an existing signing profile. This is the single client operation you’d use in build pipelines or verification actions in deployments. Swap in your registry ID and ARN for your signing profile.

$ notation sign 12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld@sha256:54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d\
    --plugin "com.amazonaws.signer.notation.plugin"\
    --id arn:aws:signer:us-west-2:12345678:/signing-profiles/image_signer\
    --force-referrers-tag=false
Successfully signed 12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld@sha256:54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d

As mentioned previously, before now we’ve used a client-side mechanism to track the referrers to an image. Historically, this method would involve the client uploading the artifact and then creating or updating an OCI image index that is tagged with the digest of the subject manifest which contained a list of all the referring artifacts to the image. Now that Amazon ECR supports OCI 1.1, you no longer need to keep pushing unnecessary tags as the registry can now track these relationships on the server. The notation CLI allows you to take advantage of this by passing along a flag --force-referrers-tag=false.

Note: For customers who began storing reference artifacts in Amazon ECR before the OCI 1.1 launch on June 27, manual cleanup of the image indexes that were pushed by 1.0 clients may be required. It is no longer necessary to use image indexes to track artifacts with the introduction of the referrers API. If you run into an error deleting an artifact or see that artifacts aren’t automatically cleaned up after the image is deleted, you may need to manually delete the image index that is no longer needed.

Client support for the referrers API is already present in many popular OCI clients. The API looks similar to the image index that was used by the client-side mechanism. You can query the API with curl to see the artifact(s) that reference the image.

$ curl --user AWS:$(aws ecr get-login-password) https://12345678.dkr.ecr.us-west-2.amazonaws.com/v2/oci-helloworld/referrers/sha256:54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d | jq .
{
  "manifests": [
    {
      "digest": "sha256:ababac8086209f68a5867de35cf4b8416275ce6236044a232a0e41c546bfb588",
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "artifactType": "application/vnd.cncf.notary.signature",
      "size": 1200,
      "annotations": {
        "com.amazonaws.signer.signingJob": "arn:aws:signer:us-west-2:12345678:/signing-jobs/70c1edfa-5ee5-4edc-9f56-b7b7f07f5f7b",
        "com.amazonaws.signer.signingProfileVersion": "arn:aws:signer:us-west-2:12345678:/signing-profiles/image_signer/GHJkuxijfA",
        "io.cncf.notary.x509chain.thumbprint#S256": "[\"d560c025ea1187950e60b53e47a7bfafbbae240512bbb8b2af0ae31f82a72acf\",\"92d7c9ed69338812010ff52b5862a332436074d1a5b69f43e12612f41219db9b\",\"eaaac975dcc0d5d160fca1e39834834f014a238cd224d053670982388ccbfca1\",\"90a87d0543c3f094dbff9589b6649affe2f3d6e0f308799be2258461c686473f\"]",
        "org.opencontainers.image.created": "2024-07-02T17:31:54Z"
      }
    }
  ],
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2
}

This demonstrates that the registry is tracking the signature artifact as a referrer to the image.

All of these details are for context. Most of the time, you’ll interact with referrers through typical workflows and not need to worry at all about the underlying details. However, if you need to investigate referrer relationships in your repositories for any reason, the ORAS project’s client has a discover command that will display the tree of artifacts that reference an image.

$ oras discover 12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld:1.0
12345678.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld@sha256:54a59583699cbd2cfef920930258449e7896038892dcc006ae31a3eb0e95f21d
└── application/vnd.cncf.notary.signature
    └── sha256:ababac8086209f68a5867de35cf4b8416275ce6236044a232a0e41c546bfb588

To wrap up the signing demonstration, we can verify the signature using Notation. The notation CLI will use the same referrers API to discover signatures and then use the signing policy associated with the registry to ensure the image matches.

$ notation verify 239609618315.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld@sha256:88c0c54329bfdc1d94d6f58cd3fcb1226d46f58670f44a8c689cb3c9b37b6925
Successfully verified signature for 239609618315.dkr.ecr.us-west-2.amazonaws.com/oci-helloworld@sha256:88c0c54329bfdc1d94d6f58cd3fcb1226d46f58670f44a8c689cb3c9b37b6925

Conclusion

We are excited to be one of the first registry cloud services to support the Referrers API and OCI 1.1. Congratulations to everyone in the community who worked together and put in many hours over the last few years to land the first major update to these specifications since launch. These features will enable future use cases for customers who are interested in storing signatures, SBOMs, scan results, and many more artifact types.

As always, we welcome feedback about this launch or other ideas for Amazon ECR at the aws/containers-roadmap, on GitHub.