Announcing software version consistency for Amazon ECS services

Note: This blog post has been updated to reflect a change on 12th September 2024. Amazon ECS no longer modifies the containerImage field found in the DescribeTask or Task Metadata output. As part of the initial ECS software version consistency release the container image tag imageUri:tag was replaced with the container image digest imageUri@digest, this is no longer the case.

Introduction

Container image tags offer a user-friendly way to manage and keep track of different versions of container images. However, they also present a security risk to organizations due to their mutable nature. Without protections in place, a container image tag can be changed in a container image repository to point to a different container image. This presents a scenario whereby the intended container image when a workload was defined may not be the one used when a workload is run.

Today we are excited to announce a new feature for Amazon Elastic Container Service (ECS): software version consistency. Amazon ECS will now resolve a container image tag to its container image digest for every version (deployment) of an Amazon ECS Service. This ensures that the same container image is used throughout the lifecycle of the deployment, and increases both the security and consistency of your applications deployed as Amazon ECS services.

Background

Amazon ECS services are a group of identical tasks used for long-running applications, commonly web and API workloads. Amazon ECS makes sure that the tasks within a service are the same by tying a version of a service to a task definition revision, called an Amazon ECS service deployment. However, when a container image tag is used within a task definition revision, it can break this consistency.

Container images are immutable. Once a container image has been built, a container image digest (a sha256 digest) is created. This image digest provides the authoritative piece of metadata for that container image, as it is created from a checksum of the container image’s contents. However, container images are not often referred to by their image digest. Instead, they are more commonly referred to by a tag.

A container image tag is not immutable, and a container image tag can change. At one point a tag may refer to one container image digest, and then at a later point in time, be updated to point to a new container image digest. This is most prevalent in the use of a “latest” container image tag. It is common to find projects using a container image tag called “latest” and regularly moving it to a new container image every time there is a new version of their software. OCI registries can implement features to prevent a container image tag from changing, such as Amazon ECR’s tag immutability. However, the use of “latest” is still widely adopted.

Software version consistency for Amazon ECS

From today onward Amazon ECS services using the Amazon ECS deployment controller use the same container image digest throughout the lifecycle of a deployment, even if the task definition revision refers to container image tags. We enforce this by capturing the image digest(s) from the first running task of a deployment, once the tags have been resolved by the container runtime. Then, we use these image digest(s) throughout the lifecycle of the deployment for all additional tasks.

The previous behavior of Amazon ECS was for each container runtime to independently resolve the image tag to a digest. Therefore, if a particular deployment frequently scaled up and down, and a container image tag was updated to point to a new container image digest, there could be different versions of a container image running under the same Amazon ECS service deployment.

For all new deployments of an Amazon ECS service, either through the creation of a new service or by updating an existing service, a container image tag is now converted and stored as a container image digest following the deployment of the first task. A walkthrough of the new deployment order for Amazon ECS services is as follows:

Amazon ECS Service

1. A new Amazon ECS service is created or updated.
2. Amazon ECS first schedules one task, regardless of the service’s desiredCount
3. Once this task is up and running, the container image digest for all of the containers in the task are captured and stored in the Amazon ECS control plane.
4. The subsequent tasks of this deployment are then scheduled, Amazon ECS leverages the stored container image digest to ensure all tasks of that deployment use the same container image.

Note: If a task definition does not use container image tags, and instead refers to each container image by it’s digest, then this new deployment order is not followed. The previous ECS behavior of scheduling all tasks at the same time is used.

Walkthrough

The software version consistency feature of Amazon ECS services can be shown by creating a new deployment of an existing service. The Amazon ECS getting started guide can help you create your first service if you do not have one. Once you have a service running, you can initiate a new service deployment with the aws ecs update-service command.

$ aws ecs update-service --cluster $CLUSTER --service $SERVICE --force-new-deployment

Once a new deployment has been created, you can verify that the new deployment pattern is being used through the DescribeService API call. In this output you can see one task is started to retrieve the container image digest, and then the subsequent tasks are started. Note that the oldest events are shown last.

$ aws ecs describe-services --cluster $CLUSTER --services $SERVICE --query 'services[0].events'
[
    {
        "id": "bf43bbc6-ad78-4316-9a8c-6b561a0c035a",
        "createdAt": "2024-06-07T14:22:31.089000+00:00",
        "message": "(service fargate-service-demo) has reached a steady state."
    },
    {
        "id": "4c3f3830-6291-47e0-a0d6-ee19f43076f8",
        "createdAt": "2024-06-07T14:22:31.088000+00:00",
        "message": "(service fargate-service-demo) (deployment ecs-svc/4506290861204002986) deployment completed."
    },
    {
        "id": "350e8c3c-890e-428c-846f-f900d964f234",
        "createdAt": "2024-06-07T14:22:13.171000+00:00",
        "message": "(service fargate-service-demo) registered 2 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:111222333444:targetgroup/defaulttgtss/74939c377ec7273a)"
    },
    {
        "id": "985ce76c-ade8-4103-9569-82dca19f95af",
        "createdAt": "2024-06-07T14:21:24.685000+00:00",
        "message": "(service fargate-service-demo) has started 2 tasks: (task 569abf2a0c1f4694882c643194278919)."
    },
    {
        "id": "290cdfd3-513a-4477-83d9-690352865bc6",
        "createdAt": "2024-06-07T14:21:18.138000+00:00",
        "message": "(service fargate-service-demo) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-west-2:111222333444:targetgroup/defaulttgtss/74939c377ec7273a)"
    },
    {
        "id": "63bfc91d-8ad6-4764-9c35-149fdfa668e9",
        "createdAt": "2024-06-07T14:20:33.898000+00:00",
        "message": "(service fargate-service-demo) has started 1 tasks: (task 08738ec750d84695ae2e12800d7a31df)."
    }
]

There are a few scenarios where a container image tag is not resolved to a container image digest, such as for services that use the CodeDeploy or External deployment controller. For a full list of caveats, see the Amazon ECS documentation.

Conclusion

With the launch of software version consistency, users can now make sure that all tasks running as part of an Amazon ECS service use the same, immutable container image. This enhancement significantly improves the reliability and security of containerized applications running on Amazon ECS. By eliminating the risk of tasks unintentionally using different container image versions, developers can have greater confidence in the consistency and predictability of their workloads. To learn more about using this feature and other capabilities of Amazon ECS, please refer to the Amazon ECS documentation. We also encourage you to share your feedback and suggestions on the AWS container services public roadmap.