Containers

Automating image compliance for Amazon ECS and Amazon EKS using Amazon Elastic Container Registry (ECR) and AWS Security Hub

Introduction

As containers move to cloud native production environments, DevOps and security teams increasingly look to deploy DevSecOps pipelines that provide automated real-time visibility into container activity, restrict container access to host and network resources and detect and prevent exploits and attacks on running containers.

In this blog post, we implement a solution that demonstrates how to use AWS Security Hub to build automated compliance for container images from the Amazon Elastic Container Registry (ECR) image repository that get deployed either on Amazon Elastic Container Service or Amazon Elastic Kubernetes Service (EKS) clusters. Our solution automatically generates an AWS Security Hub finding whenever a vulnerable image that contains Critical or High Common Vulnerabilities and Exposures (CVE) is pushed to ECR. The finding can then be triaged and remediated from AWS Security Hub based on the image scan results. Our solution then implements an automated AWS Security Hub remediation action to allow or block the image from being instantiated into the container orchestrator. The entire solution deploys on AWS with 1-click automation using AWS CloudFormation.

Key benefits of using Amazon ECR with AWS Security Hub for automating Amazon ECS and Amazon EKS image compliance

If you are developing modern applications, then it is likely that you are are using containers. A container image is a standard way to package your application’s dependencies, code and configuration into a single Docker image. These Docker container images need to be scanned for vulnerabilities. It is paramount that images kept in AWS Elastic Container Registry (ECR) are clean from vulnerabilities. These vulnerabilities are classified as MEDIUM, HIGH, or CRITICAL when an image is scanned by ECR either via a scan on push or a manual scan. From here, the Common Vulnerabilities and Exposures (CVEs) findings are pushed to AWS Security Hub.  Our AWS Security Hub based remediation action restricts access to any ECR container image when a Critical or High vulnerability is detected during an image scan. This allows a DevOps person to take proper remediation action on the container image. For example, if a HIGH CVE is found as shown in the NGINX image below, it will be automatically blocked from being deployed to the EKS or ECS cluster and thus ensuring proper image compliance.

Prerequisites

  1. You must have Security Hub enabled in the AWS Region where you deploy this solution.  Follow the steps documented here to enable AWS Security Hub in your AWS account or in your AWS Organization.

Solution overview

This automated image compliance solution for Amazon EKS and Amazon ECS is based on automation of the detection and remediation of Amazon ECR Image Scan events using AWS CloudWatch events and AWS Security Hub Custom Actions. It consists of one AWS CloudFormation template that fully automates the provisioning, setup, and integration of all the components necessary for this solution.

The following architecture illustrates the components of our solution and how they integrate with each other. As shown in the architecture, the solution generates an AWS Security Hub finding whenever a vulnerable image is scanned in ECR. This is either when an ECR repository is configured for a scan on push or via a manual scan of the image. The solution provisions an Amazon CloudWatch Events (EventBridge) Rule that gets triggered based on a Amazon ECR Event for a completed image scan. The target for the Amazon CloudWatch Events (EventBridge) Rule is an AWS Lambda function that translates the event from the image scan into an AWS Security Finding Format for AWS Security Hub. Our solution also provisions an AWS Security Hub Custom Action for remediation. The Security Hub based remediation attaches an Amazon ECR Repository policy as soon the remediation action is invoked, and the policy is scoped for controlling access to the specific repository where the vulnerable image is detected.

Set up the solution and test

Set up

This solution for ECR image compliance with Security Hub is available for download here with a detailed readme.  AWS CloudFormation fully automates the set up for this solution in 1 step:

Navigate to the AWS CloudFormation in the AWS Management Console and launch the aws-ecr-continuouscompliance-v1.yml template. The template takes no parameters. The following components are provisioned by this template:

    1. Amazon CloudWatch Events (EventBridge) Rule:
      1. The CloudWatch Events Rule is triggered based on an ECR event for a completed image scan
    2. AWS Lambda as a target for the CloudWatch Events Rule:
      1. Obtains event details from the ECR completed image scan event.
      2. Sends finding to Security Hub via ASFF
    3. AWS Security Hub based remediation
      1. Creates an Amazon CloudWatch Events Rule, which is triggered based on a AWS Security Hub Custom Action
      2. Provisions an AWS Lambda as a target for the AWS Security Hub Custom Action
      3. AWS Lambda creates an ECR Repository policy that denies access if the image scan event has a vulnerability (Critical or High)

Test

  1. Follow the steps as outlined here to push an image with known vulnerabilities to ECR (e.g. nginx:latest).
  2. Navigate to AWS Security Hub in the AWS console and click on Findings in the left panel. Select the relevant finding and with our solution you can also search for ECR related findings by adding a filter with Resource Type is AwsEcr in the top panel. Refer to the following screenshot below:
  3. With the relevant finding selected in the Findings panel, select Actions from the top of the panel and click on the ‘ECR1’ action. Refer to the following screenshot below:
  4. Navigate to Amazon ECR in the AWS Management Console, select the repository that contains the vulnerable image, and validate the deny permissions policy provisioned by the Security Hub remediation action by selecting Permissions in the left panel.

We will assume that you already have an ECS or EKS cluster up and running to test that the ECR image will be blocked from deploying on the ECS or EKS cluster.

We will perform a simple test on an EKS cluster to demonstrate this but you can also perform similar tests with ECS. You can follow this workshop here to get started with Amazon EKS and the use of eksctl and kubectl. In the following steps we will try to run the nginx:latest image from the ECR repository:

kubectl run mynginx --image=1234567890.dkr.ecr.us-east-1.amazonaws.com/nginx:latest
—restart=Never

kubectl get po mynginx

NAME READY STATUS RESTARTS AGE
mynginx 0/1 ImagePullBackOff 0 17s

As you can see the pod is in an ImagePullBackoff state. When we go to describe the pod, we see the following error due to the deny policy set from the remediation action on Security Hub.

kubectl describe po mynginx

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/mynginx to ip-10-0-124-36.us-east-2.compute.internal
Normal Pulling 17s (x2 over 32s) kubelet, ip-10-0-124-36.us-east-2.compute.internal Pulling image "1234567890.dkr.ecr.us-east-1.amazonaws.com/nginx:latest"
Warning Failed 17s (x2 over 32s) kubelet, ip-10-0-124-36.us-east-2.compute.internal Failed to pull image "1234567890.dkr.ecr.us-east-1.amazonaws.com/nginx:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for 1234567890.dkr.ecr.us-east-1.amazonaws.com/nginx, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::1234567890:assumed-role/ClusterStack-us-east-2-octankfintechclusterNodegro-10G5B4K6ECV6C/i-0b219fe9c618ba23d is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:us-east-1:1234567890:repository/nginx with an explicit deny
Warning Failed 17s (x2 over 32s) kubelet, ip-10-0-124-36.us-east-2.compute.internal Error: ErrImagePull
Normal BackOff 3s (x2 over 31s) kubelet, ip-10-0-124-36.us-east-2.compute.internal Back-off pulling image "1234567890.dkr.ecr.us-east-1.amazonaws.com/nginx:latest"
Warning Failed 3s (x2 over 31s) kubelet, ip-10-0-124-36.us-east-2.compute.internal Error: ImagePullBackOff

See the events section from the above command, which clearly blocked the image (arn:aws:ecr:us-east-1:1234567890:repository/nginx with an explicit deny) from being deployed due to pull access denied on this particular image due to Critical or High CVE

Conclusion

In this blog post, we have demonstrated a cloud native solution in AWS for image compliance for container images deployed in ECS or EKS clusters by automating the detection and remediation of vulnerabilities of scanned images using Amazon ECR and AWS Security Hub. This ensures that only approved images are deployed to our EKS clusters. It also enables centralized triaging and remediation of vulnerabilities in these container images using AWS Security Hub. We hope that you found this solution helpful and look forward to your feedback.