Automating custom Amazon EKS worker node builds using EC2 Image Builder

Customers who are building their “Golden Image” Amazon Machine Images (AMIs) using EC2 Image Builder may wish to extend their Image Builder pipelines to build out their Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes as well. In this blog, we will show you how to do this and provide you with AWS CloudFormation templates to get you started.

In this post, we will use the concept of Image Builder Cascading Pipelines. The image created from the golden image pipeline will be the source image of the EKS worker node pipeline. Image Builder can run pipelines on a recurring schedule when dependencies, such as base images or components, are updated. Additionally, the custom EKS build process incorporates the latest Kubernetes binary patches. By combining these two processes, new EKS worker nodes will always be built with the latest EC2 image builds and Kubernetes patches.

Figure 1. EC2 Image Builder Cascading Pipeline workflow

Walkthrough

In the walkthrough, we will build an EKS worker node from our organization’s golden image pipeline. You don’t have to build from an existing Image Builder pipeline, but this would be the most common use case for implementing this solution.

The high-level process is:

• Create an EC2 Instance Profile with necessary permissions.
• Create a new Image Builder Component with Linux commands to install EKS required binaries.
• Create a new Image Builder Recipe with the golden image pipeline image as the base image.
• Create a new Image Builder Pipeline that runs weekly to build new EKS worker node AMIs.

Note: We will be going through each step manually in the AWS console to help with understanding, but we recommend leveraging the CloudFormation templates referenced at the bottom of the post, with any necessary modifications, for actual use.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• Basic knowledge of Image Builder
• A working Amazon EKS cluster
• AWS account access with permissions to:
  • Create new Identity and Access Management (IAM) policies and roles
  • Create Image Builder resources

Creating an EC2 Instance Profile

Before we can get started with the Image Builder process, we must first create an IAM role that will serve as an EC2 Instance Profile during the actual build. For instructions on creating an IAM role for an AWS service, please follow the official AWS documentation. The role should have the following policies attached:

• AWS Managed AmazonSSMManagedInstanceCore policy
• AWS Managed EC2InstanceProfileForlmageBuilder policy
• Custom IAM policy with permissions to read from the AWS managed public “amazon-eks” S3 bucket (listed below)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::amazon-eks/*",
        "arn:aws:s3:::amazon-eks"
      ],
      "Effect": "Allow"
    }
  ]
}

Figure 2. EC2 Image Builder EC2 Instance Profile permissions

Note: If you are operating in a different partition than the default AWS partition, such as AWS GovCloud, the Amazon Resource Name (ARN) will begin with a different value than “arn:aws”. This is still true even though the official Amazon managed EKS binaries are hosted in the AWS Commercial partition. More information can be found on ARNs here. GovCloud example:

"arn:aws-us-gov:s3:::amazon-eks/*",
"arn:aws-us-gov:s3:::amazon-eks"

To serve as an instance profile, the role must also have a trust policy that allows the EC2 service to assume the role. The trust policy should look like this:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }
}

Figure 3. EC2 Image Builder EC2 Instance Profile trust policy

Creating an Image Builder Component

1. Open the Image Builder console.
2. In the navigation pane, choose Components.
3. Choose Create component.
4. Leave the Type as Build.
5. Select the Image operating system and Compatible OS Versions you would like to use. Compatible OS Versions include:
   • Amazon Linux 2023
   • Red Hat Enterprise Linux 8 and 9
6. Specify a KMS key (optional), Component version, and description (optional).
7. In the Content section, type in your Component yaml or paste in the yaml from the link you selected above.

Figure 4. EC2 Image Builder Component definition

Creating an Image Builder Recipe

Next, we will create a new Image Builder Recipe that will execute the component we just created.

1. Remain in the Image Builder console.
2. In the navigation pane, choose Image recipes.
3. Choose Create image recipe.
4. Enter a Name, Version, and Description (optional).

Figure 5. EC2 Image Builder recipe creation

5. Select the Image Operating System (OS) and Image origin you would like to use.
   • To use your organization’s golden image, select Images owned by me or Images shared with me under Image origin. Otherwise, select Quick start.
   • We are using the Red Hat Enterprise Linux (RHEL) OS.
6. Select your Image name from the populated list.
   • We are selecting a DISA STIG compliant RHEL image we built using Image Builder.

Figure 6. EC2 Image Builder recipe Operating System selection

7. The Instance configuration and Working directory sections don’t require any changes.
8. In the Components section, select the Build component you created in the previous section, along with any others you would like. You will need to make sure your Filter is set to Owned by me to find your component.
   • The EKS worker node installation scripts require the existence of the AWS Command Line Interface (AWS CLI). If your image doesn’t already have the CLI installed, you can have the Amazon-Managed Build component install it for you before executing the EKS Component as shown in Figure 7 below.

Figure 7. EC2 Image Builder recipe build component selection

9. (Optionally) select any Test components you would like to run during the build process.
   • We are executing a simple reboot Test component.

Figure 8. EC2 Image Builder recipe test component selection

10. (Optionally) modify the EBS volumes of your build instance and choose Create recipe.

Figure 9. EC2 Image Builder recipe storage configuration

Creating an Image Builder Pipeline

Now that we have created a Component and Recipe, we are ready to create an Image Builder Pipeline that will run on a schedule to build our EKS worker node images.

1. Remain in the Image Builder console.
2. In the navigation pane, choose Image pipelines and choose Create image pipeline.
3. Enter a Pipeline name and Description (optional)
4. Make sure Enhanced metadata is unchecked unless you have already configured inventory collection through AWS Systems Manager in your account.
   • If you leave this option checked and have not configured inventory collection, the pipeline will hang and ultimately fail.
5. Choose where or not to enable EC2 image scanning and ECR container scanning through Amazon Inspector.
6. Choose a Build schedule and choose
   • If using a schedule, we recommend selecting the option to Run pipeline at the scheduled time if there are dependency updates.
7. Under Choose recipe, select Use existing recipe, and select the Recipe created in the previous section and choose Next.
8. Under Define image creation process, leave the Default workflows options selected and choose Next.
9. Under Define infrastructure configuration, select Create a new infrastructure configuration.
   • Give the infrastructure configuration a Name, Description (optional), and select the IAM role you created at the beginning of the walkthrough.
   • No other infrastructure configurations should be required unless you need to specify other settings like a VPC configuration for the build instance.
10. Choose Next.
11. Select any Distribution settings you would like and choose Next.
12. Confirm all settings and choose Create pipeline.

Figure 10. EC2 Image Builder pipeline summary

Running an Image Builder Pipeline

Our Image Builder pipeline is now configured and ready for execution.

1. Remain in the Image Builder console.
2. In the navigation pane, choose Image pipelines.
3. Select the pipeline you just created.
4. Choose Actions and choose Run pipeline.

Validating the EKS worker node build

At this point our pipeline build process is under way, and we want to ensure the build process completes successfully.

1. Remain in the Image Builder console.
2. In the navigation pane, choose Image pipelines.
3. Select the pipeline you just executed.
4. Under the Log stream column, select the link to view the CloudWatch Log stream.

Figure 11. EC2 Image Builder CloudWatch Logs successful messages

We want to follow the progress of our Image Builder pipeline execution and ensure there are no errors that cause the pipeline to fail. The process can take anywhere from 10 minutes to an hour depending on the environment and pipeline configuration. Ultimately, we should see a message stating our image has transitioned to an AVAILABLE state

All Workflows ran successfully for Image ARN: arn:aws:imagebuilder:us-east-1:123456789012:image/rhel-eks-sitg-recipe/1.29.0/1
Image transitioned to DISTRIBUTING state for Image ARN: arn:aws:imagebuilder:us-east-1:123456789012:image/rhel-eks-stig-recipe/1.29.0/1
Image transitioned to INTEGRATING state for Image ARN: arn:aws:imagebuilder:us-east-1:123456789012:image/rhel-eks-stig-recipe/1.29.0/1
Image transitioned to AVAILABLE state for Image ARN: arn:aws:imagebuilder:us-east-1:123456789012:image/rhel-eks-stig-recipe/1.29.0/1

EKS worker node deployment

The EKS worker node build process is now complete, and we can now create EKS node groups using our new AMI and join them to our cluster. For information on this process, please review the official EKS documentation.

CloudFormation Templates

We walked through each step manually to help with understanding the process, but using CloudFormation to build out these Image Builder constructs is the recommended approach. Below you will find links to example CloudFormation templates for each of the operating systems mentioned earlier in the walkthrough. Feel free to copy them and modify them to fit your use case and environment.

• Amazon Linux 2023
• Red Hat Enterprise Linux 8 and 9

Cleaning up

If you went through each step of the walkthrough manually, delete each Image Builder construct and any resulting AMIs to avoid incurring future charges. You will need to delete the Image Builder constructs in reverse order of creation. If you deployed this solution using CloudFormation, delete the CloudFormation stack and any resulting AMIs to avoid incurring future charges.

Conclusion

In this post, we showed you how to create an EC2 Image Builder Pipeline that will build Amazon EKS worker nodes on a recurring schedule based off images from an organization’s golden image pipeline. Using this approach should ensure your EKS worker nodes always have your organization’s latest patches and software versions as well as the latest versions of EKS binaries. As you begin automating your AMI builds, you can implement Image Builder lifecycle management policies to cleanup outdated images and control costs.