Containers

Getting started with Bottlerocket and Amazon ECS

Last week we announced the general availability of the Amazon Elastic Container Service (Amazon ECS)-optimized Bottlerocket AMI and Bottlerocket support for Amazon ECS is now generally available. Bottlerocket is an open source project that focuses on security and maintainability, providing a reliable, and consistent Linux distribution for hosting container-based workloads.

In this post, I am going to show you how to use Amazon ECS and Bottlerocket together to run your applications in a safe and secure manner.

Why Bottlerocket?

Customers have continued to adopt containers to run their workloads, and AWS saw a need for a Linux distribution designed and optimized to run these containerized applications. Bottlerocket OS was built to provide a secure foundation for hosts running containers, and minimizing operational overhead to manage them at scale.

Security

The root filesystem in Bottlerocket is marked as read-only and cannot be directly modified by userspace processes. Bottlerocket uses dm-verity for its root filesystem image (dm-verity is a Linux kernel module that provides transparent integrity checking of block devices using a cryptographic digest to check for tampering). This protects against some container escape vulnerabilities such as CVE-2019-5736. The kernel is configured to restart if corruption is detected. This allows the system to fail closed if the underlying block device is unexpectedly modified.

A user cannot modify system configuration files such as /etc/resolv.conf or /etc/containerd/config.toml directly. Modifications are made through the API. The API runs locally on the instance as an HTTP interface and is the primary way to read and modify operating system settings, to update services based on those settings, and to learn about and change the state of the operating system.

These settings are persisted across reboot and migrated through OS upgrades. They are used to render system configuration files from templates on every boot. The executables that are available in Bottlerocket are built with hardening flags, which makes addresses harder to predict for an attacker if they attempt to exploit a memory corruption vulnerability.

And lastly, Bottlerocket enables SELinux by default, sets it to enforcing mode, and loads the policy during boot. There is no way to disable it.

Simplified Operational Tasks

Bottlerocket is designed for reliable updates that can be applied through automation.

This is achieved through the following mechanisms:

  • Two partition sets that use an active/passive flip to swap OS images.
  • Declarative API with modeled settings for runtime configuration.

Upgrading an instance running Bottlerocket is similar to a firmware upgrade on a physical device or applying an operating system update to your phone. You don’t need to install a bunch of software packages to get the new version. You download a full system disk image and apply the update, which is written to alternate partition set. When you are ready to activate the upgrade, you reboot the instance which now boots from the new version primary partition with a new version of the OS. All the data and persistent configuration is stored on a separate data partition and is available after the reboot.

User interaction with Bottlerocket

As a user, you will not have direct access to the underlying operating system (by default you will not be able to SSH into the instance) but you will rather interact with two different containers that will provide certain interfaces into the operating system.

Bottlerocket has a “control” container, enabled by default, that runs outside of the orchestrator in a separate instance of containerd. This container runs the AWS SSM agent that allows you to run commands or start shell sessions on Bottlerocket instances in EC2 (you can replace this control container with your own, refer to the documentation for more info).

In order to access the container through SSM, you need to give your instance the appropriate IAM role for this to work (I will show you how to do this later on in this post).

Once you have access to the control container, you can execute the commands, which make the appropriate API calls to a local service running on the instance to configure and manage your Bottlerocket instance.

This is not a fully fledged shell environment and you have a limited set of commands available. Most of your interactions will be through the apiclient command. You can find more details in the documentation.

Bottlerocket also has an “administrative” container, which is disabled by default and runs in a different process of containerd. This container has an SSH server running, that lets you log in as ec2-user using your EC2-registered SSH key.

Bottlerocket uses two separate copies of a container runtimes in order to isolate orchestrator-driven workloads (i.e., customer workloads managed through ECS or through Kubernetes) from system workloads ( the “admin” and “control” containers). This helps reduce the blast radius of possible problems with the orchestrated workloads and keep the underlying system functional. In the case of the ECS variant, Docker is used for orchestrated containers and containerd is used for the host containers.

There are two ways to enable the administrative container, you can configure the setting in user data when launching the instance through EC2 in the AWS Management Console or AWS CLI.

[settings.host-containers.admin]
enabled = true

You can also enable the admin container from within the control container by running the following command from the local shell:

enable-admin-container

Once the container is active, you can access the instance through SSH with your private key that matches the key pair that you provided when launching the instance. When you SSH into a Bottlerocket instance, you land in a highly privileged Amazon Linux 2 container. The container has the host filesystem mounted at .bottlerocket/root and you have the full ecosystem of AL2 packages at your disposal. This container also runs in the host pid namespace (among other namespaces it shares with the host) which means commands like ps will show you the host’s process list. sheltie lets you jump into the host’s other namespaces and uses a static bash shell to let you get even closer to the host.

I mentioned an ECS variant above and I think it is worthwhile to go into a bit more detail about variants. A variant is a list of only the necessary software installed, plus a model that defines the API. We want to keep the footprint of Bottlerocket as small as possible for security and performance reasons, so there are different variants that are customized for different use cases, each with its own set of software and API settings. If you would like to learn more about variants and how to build them, please refer to the Bottlerocket documentation.

Setting up an ECS cluster with Bottlerocket

Let’s have a look how this is done in practice. I will be working in the us-west-2 (Oregon) region.

Prerequisites

  • The AWS CLI with appropriate credentials
  • A default VPC in a region of your choice (you can also use an existing VPC in your account)
  • A key pair in your account for remote access (my key pair is named ecs-botterocket)

First, I will create the ECS cluster named ecs-bottlerocket

aws ecs --region us-west-2 create-cluster --cluster-name ecs-bottlerocket

The instances we are going to launch are going to need an IAM role in order to communicate both with the ECS APIs and the Systems Manager Session Manager APIs as well. I have created an IAM role named ecsInstanceRole that has both the AmazonSSMManagedInstanceCore and the AmazonEC2ContainerServiceforEC2Role managed policies attached.

The list of AMIs is publicly available from AWS Systems Manager Parameter Store, so let’s get the AMI ID for the latest Bottlerocket release.

latest_bottlerocket_ami=$(aws ssm get-parameter --region us-west-2 \
    --name "/aws/service/bottlerocket/aws-ecs-1/x86_64/latest/image_id" \
    --query Parameter.Value --output text)

First, let’s get the ID of the default VPC.

vpc_id=$(aws ec2 describe-vpcs \
   --region us-west-2 \
   --filters=Name=isDefault,Values=true \
   --query 'Vpcs[].VpcId' --output text)

Next, we get the list of subnets that are configured to allocate a public IP address.

aws ec2 describe-subnets \
   --region us-west-2 \
   --filter=Name=vpc-id,Values=$vpc_id \
   --query 'Subnets[?MapPublicIpOnLaunch == `true`].SubnetId'
   
[
    "subnet-bc8993e6",
    "subnet-b55f6bfe",
    "subnet-e1e27fca",
    "subnet-21cbc058"
]

In order to associate our EC2 instances to the ECS cluster, I need to provide some information to the instance when I create it: a small config file that has the details of my ECS cluster that I will save in a file my current directory. A full set of supported settings is here.

cat > ./userdata.toml << 'EOF'
[settings.ecs]
cluster = "ecs-bottlerocket"
EOF

I am going deploy two Bottlerocket instances, one each in two of the subnets above, I am going to choose two public subnets for this blog post. It will be easier to debug and connect to the instances if needed. You can choose private or public subnets based on your use case.

aws ec2 run-instances --key-name ecs-bottlerocket \
   --subnet-id subnet-bc8993e6 \
   --image-id $latest_bottlerocket_ami \
   --instance-type t3.large \
   --region us-west-2 \
   --tag-specifications 'ResourceType=instance,Tags=[{Key=bottlerocket,Value=quickstart}]' \
   --user-data file://userdata.toml \
   --iam-instance-profile Name=ecsInstanceRole
   
aws ec2 run-instances --key-name ecs-bottlerocket \
   --subnet-id subnet-b55f6bfe \
   --image-id $latest_bottlerocket_ami \
   --instance-type t3.large \
   --region us-west-2 \
   --tag-specifications 'ResourceType=instance,Tags=[{Key=bottlerocket,Value=quickstart}]' \
   --user-data file://userdata.toml \
   --iam-instance-profile Name=ecsInstanceRole

I want to check the status of the environment. First, let’s look at the cluster. Here you can see that it is active and running two instances.

aws ecs describe-clusters --clusters ecs-bottlerocket --region us-west-2

{
    "clusters": [
        {
            "clusterArn": "arn:aws:ecs:us-west-2:123456789012:cluster/ecs-bottlerocket",
            "clusterName": "ecs-bottlerocket",
            "status": "ACTIVE",
            "registeredContainerInstancesCount": 2,
            "runningTasksCount": 0,
            "pendingTasksCount": 0,
            "activeServicesCount": 0,
            "statistics": [],
            "tags": [],
            "settings": [
                {
                    "name": "containerInsights",
                    "value": "disabled"
                }
            ],
            "capacityProviders": [],
            "defaultCapacityProviderStrategy": []
        }
    ],
    "failures": []
}

Let’s look at the container instances to see if they are running Bottlerocket. Below, you can see that I have two running.

aws ecs list-container-instances --cluster ecs-bottlerocket --region us-west-2

{
    "containerInstanceArns": [
        "arn:aws:ecs:us-west-2:123456789012:container-instance/ecs-bottlerocket/4061b1612a8b479695c638e9890136f4",
        "arn:aws:ecs:us-west-2:123456789012:container-instance/ecs-bottlerocket/8e06130837cf445aab0a79a82a363894"
    ]
}
    
aws ecs describe-container-instances \
   --region us-west-2 \
   --container-instances 4061b1612a8b479695c638e9890136f4 8e06130837cf445aab0a79a82a363894  \
   --cluster ecs-bottlerocket \
   --query 'containerInstances[].attributes[?name ==`bottlerocket.variant`]'

[
    [
        {
            "name": "bottlerocket.variant",
            "value": "aws-ecs-1"
        }
    ],
    [
        {
            "name": "bottlerocket.variant",
            "value": "aws-ecs-1"
        }
    ]
]

You can see that the instances have the attribute bottlerocket-variant with a value of aws-ecs-1, this is Bottlerocket variant we mentioned above that is customized specifically with the minimal amount of packages for ECS.

Next, I will create a task definition for the sample application.

cat > ./sample-task.json << 'EOF'
{
    "family": "ecs-bottlerocket-sample", 
    "networkMode": "awsvpc", 
    "containerDefinitions": [
        {
            "name": "sample-app", 
            "image": "httpd:2.4", 
            "portMappings": [
                {
                    "containerPort": 80, 
                    "hostPort": 80, 
                    "protocol": "tcp"
                }
            ], 
            "essential": true, 
            "entryPoint": [
                "sh",
        "-c"
            ], 
            "command": [
                "/bin/sh -c \"echo '<html> <head> <title>Amazon ECS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center> <h1>Amazon ECS Sample App</h1> <h2>Congratulations!</h2> <p>Your application is now running on a container in Amazon ECS.</p> </div></body></html>' >  /usr/local/apache2/htdocs/index.html && httpd-foreground\""
            ]
        }
    ], 
    "requiresCompatibilities": [
        "EC2"
    ], 
    "cpu": "256", 
    "memory": "512"
}
EOF

Register the task in ECS.

aws ecs register-task-definition \
   --region us-west-2 \
   --cli-input-json file://sample-task.json

Create the service on my ECS cluster.

aws ecs create-service --cluster ecs-bottlerocket \
   --region us-west-2 \
   --service-name bottlerocket-service \
   --task-definition ecs-bottlerocket-sample:1 \
   --desired-count 4 --launch-type "EC2" \
   --network-configuration "awsvpcConfiguration={subnets=[subnet-bc8993e6,subnet-b55f6bfe]}"

Finally, let’s check the status of my service. We will see four running tasks.

aws ecs describe-services --cluster ecs-bottlerocket \
   --service bottlerocket-service \
   --region us-west-2 \
   --query 'services[].deployments'
   
[
    [
        {
            "id": "ecs-svc/9699136747523902404",
            "status": "PRIMARY",
            "taskDefinition": "arn:aws:ecs:us-west-2:123456789012:task-definition/ecs-bottlerocket-sample:1",
            "desiredCount": 4,
            "pendingCount": 0,
            "runningCount": 4,
            "failedTasks": 0,
            "createdAt": "2021-06-30T11:26:16.405000+03:00",
            "updatedAt": "2021-06-30T11:26:46.746000+03:00",
            "launchType": "EC2",
            "networkConfiguration": {
                "awsvpcConfiguration": {
                    "subnets": [
                        "subnet-bc8993e6",
                        "subnet-b55f6bfe"
                    ],
                    "securityGroups": [],
                    "assignPublicIp": "DISABLED"
                }
            },
            "rolloutState": "COMPLETED",
            "rolloutStateReason": "ECS deployment ecs-svc/9699136747523902404 completed."
        }
    ]
]

Cleanup

To remove the resources that you created during this post, run the following commands.

aws ecs update-service --cluster ecs-bottlerocket \
   --region us-west-2 \
   --service bottlerocket-service \
   --desired-count 0
  
aws ecs delete-service --cluster ecs-bottlerocket \
   --region us-west-2 \
   --service bottlerocket-service
  
aws ecs deregister-task-definition \
   --region us-west-2 \
   --task-definition ecs-bottlerocket-sample:1
   
delete_instances=$(aws ec2 describe-instances --region us-west-2 \
   --filters "Name=tag-key,Values=bottlerocket" "Name=tag-value,Values=quickstart" \
   --query 'Reservations[].Instances[].InstanceId')  
   
for instance in $delete_instances
  do aws ec2 terminate-instances --instance-ids $instance --region us-west-2
done 

aws ecs delete-cluster \
   --region us-west-2 \
   --cluster ecs-bottlerocket

Conclusion

In this post, we discussed what Bottlerocket is, how it works, and how you can interact directly with the underlying operating system. We also deployed an ECS service running on Bottlerocket as the underlying operating system. As you can see, there is no difference in the way you deploy your ECS tasks and services onto Bottlerocket from other operating systems.

Bottlerocket’s components are open-source. Our intent is for Bottlerocket to be a collaborative community project, so you have the ability to contribute directly and make your own customized versions. I would like to welcome you to get involved with Bottlerocket and provide your feedback. Check out our GitHub repository for discussion via issues and contributions via pull requests. We also have the #bottlerocket channel for informal interaction in the AWS Developer Slack; you can sign up here.

TAGS: ,
Maish Saidel-Keesing

Maish Saidel-Keesing

Maish Saidel-Keesing is a Senior Enterprise Developer Advocate in the Container Services team. Maish lives in Israel with his wife and 3 daughters and focuses on improving customer experience with everything related to containers in the cloud. You can always reach out to him on Twitter (@maishsk).