Introduction
As Kubernetes adoption grows, many organizations are choosing it as their platform to build and host their modern and secure applications. Security is one of the primary design criteria for many workloads, especially those dealing with sensitive data such as financial data processing. These workloads have a stringent requirement to adhere to various security and compliance controls.
Many Amazon Elastic Kubernetes Service (Amazon EKS) customers, especially enterprise customers from the Banking and Finance industry, are looking for guidance from AWS on hardening Amazon EKS.
The Center for Internet Security (CIS) Benchmarks are best practices for the secure configuration of a target system. They define various Benchmarks for the Kubernetes control plane and the data plane. For Amazon EKS clusters, it’s strongly recommended to follow the CIS Amazon EKS Benchmark. However, many organizations also need to harden the operating system on the worker nodes for security and compliance purposes. Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for containers. If an organization needs to ensure compliance, the organization must implement the CIS Benchmark for Bottlerocket. This post provides detailed, step-by-step instructions on how customers can bootstrap an Amazon EKS optimized Bottlerocket Amazon Machine Image (AMI) for the requirements of the CIS Bottlerocket Benchmarks. This post also illustrates how to continuously validate the worker nodes against the Benchmark after deployment to minimize the risk of security configuration drift.
Amazon EKS optimized Bottlerocket AMI hardening process
The CIS Bottlerocket Benchmark defines two profiles for hardening (i.e., Level 1 and Level 2):
- A Level 1 profile is intended to be practical and prudent, provide a clear security benefit, and not inhibit the utility of the technology beyond acceptable means.
- A Level 2 profile is intended for environments or use cases where security is paramount, acts as a defense in depth measure, and may negatively inhibit the utility or performance of the technology.
Solution overview
The following walkthrough is a solution for hardening and validating an Amazon EKS optimized Bottlerocket AMI against Level 2.
Amazon EKS optimized Bottlerocket AMI support for CIS Benchmark
The Amazon EKS optimized Bottlerocket AMI (as of this writing) supports 18 out of 28 Level 1 and 2 recommendations specified in the CIS Benchmark for Bottlerocket, without a need for any additional configuration effort. For the remaining 10 recommendations to adhere to Level 2, six recommendations can be addressed via a bootstrap container and four recommendations can be addressed via kernel sysctl configurations in the user data of the Amazon EKS worker nodes.
| Section Number |
Section |
Level 1 |
Level 2 |
Total |
| 1 |
Initial setup |
7 |
3 |
10 |
| 2 |
Services |
1 |
0 |
1 |
| 3 |
Network configuration |
3 |
12 |
15 |
| 4 |
Logging and auditing |
2 |
0 |
2 |
|
|
13 |
15 |
28 |
Walkthrough
The code used in this solution is available in GitHub. Please clone the repository to prepare for the walkthrough.
git clone https://github.com/aws-samples/containers-blog-maelstrom.git
cd containers-blog-maelstrom/cis-bottlerocket-benchmark-eks/
Prerequisites
You’ll need the following to complete the tutorial:
Note: We have tested the CLI steps in this post on Amazon Linux 2.
You’ll also need to configure the following environment variables:
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
export CLUSTER_NAME=cis-bottlerocket
export BOOTSTRAP_ECR_REPO=bottlerocket-cis-bootstrap-image
export VALIDATION_ECR_REPO=bottlerocket-cis-validation-image
Building a bootstrap container image
Bootstrap containers on Bottlerocket are specialized host containers that run on a new node as it is launched. Bootstrap containers are used to configure host OS settings that are not available via the Bottlerocket API. For the CIS Benchmark for Bottlerocket, a bootstrap container is needed to update the iptables rules of the host to be in compliance with Level 2 of the CIS Benchmark. The bootstrap container’s Entrypoint executes the following bash script, which, aside from the rules allowing inbound kubelet traffic to the instance, is sourced directly from the Benchmark documentation.
bottlerocket-cis-bootstrap-image/bootstrap-script.sh
#!/usr/bin/env bash
# Flush iptables rules
iptables -F
# 3.4.1.1 Ensure IPv4 default deny firewall policy (Automated)
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
# Allow inbound traffic for kubelet (so kubectl logs/exec works)
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
# 3.4.1.2 Ensure IPv4 loopback traffic is configured (Automated)
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -s 127.0.0.0/8 -j DROP
# 3.4.1.3 Ensure IPv4 outbound and established connections are configured (Manual)
iptables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p icmp -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -m state --state ESTABLISHED -j ACCEPT
# Flush ip6tables rules
ip6tables -F
# 3.4.2.1 Ensure IPv6 default deny firewall policy (Automated)
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP
# Allow inbound traffic for kubelet on ipv6 if needed (so kubectl logs/exec works)
ip6tables -A INPUT -p tcp --destination-port 10250 -j ACCEPT
# 3.4.2.2 Ensure IPv6 loopback traffic is configured (Automated)
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o lo -j ACCEPT
ip6tables -A INPUT -s ::1 -j DROP
# 3.4.2.3 Ensure IPv6 outbound and established connections are configured (Manual)
ip6tables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
ip6tables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
ip6tables -A OUTPUT -p icmp -m state --state NEW,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p icmp -m state --state ESTABLISHED -j ACCEPT
For our walkthrough, we’ll use a script from the repository to create an Amazon Elastic Container Registry (Amazon ECR) repository to store the container image. You can safely ignore the error message An error occurred (RepositoryNotFoundException) output while running the script.
cd bottlerocket-cis-bootstrap-image
chmod +x create-ecr-repo.sh
./create-ecr-repo.sh
Next, let’s build the bootstrap container image and push it to the newly created Amazon ECR repository.
The output from the make command should look like the following code:
Building the docker image: bottlerocket-cis-bootstrap-image-t:latest using bottlerocket-cis-bootstrap-image-t/Dockerfile...
[+] Building 3.3s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 226B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.7s
=> CACHED [1/4] FROM docker.io/library/alpine@sha256:b95359c2505145f16c6aa384f9cc74eeff78eb36d308ca4fd902eeeb0a0b161b 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 2.10kB 0.0s
=> [2/4] COPY ./bootstrap-script.sh / 0.0s
=> [3/4] RUN chmod +x /bootstrap-script.sh 0.4s
=> [4/4] RUN apk update && apk add bash && apk add iptables && apk add ip6tables 2.0s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:e1bb2264fd429e4b47f79af6035684b1ebc34e6fcd9b784c540ec16a56baaccb 0.0s
=> => naming to XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/bottlerocket-cis-bootstrap-image-t:latest 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
#@docker build --no-cache -f ./bottlerocket-cis-bootstrap-image-t/Dockerfile -t XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/bottlerocket-cis-bootstrap-image-t:latest
Pushing the docker image for XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/bottlerocket-cis-bootstrap-image-t:latest ...
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/bottlerocket-cis-bootstrap-image-t
Login Succeeded
The push refers to repository [XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/bottlerocket-cis-bootstrap-image-t]
132f2a14454d: Pushed
fbadbb73a582: Pushed
3fa058784363: Pushed
e5e13b0c77cb: Pushed
latest: digest: sha256:97856f5c9ec99320b6009a8f37389ce91086d3fbf815301abc4e81a08f1c43b2 size: 1153
Create Amazon EKS cluster with Amazon EKS-managed node group with Bottlerocket AMI
With the bootstrap container created and ready for use in Amazon ECR, we can create an Amazon EKS cluster with a managed node group running Bottlerocket configured to CIS Bottlerocket Benchmark. We’ll use the following cat command to insert the environment variables defined earlier into the cluster.yaml file located in the root of the GitHub repository.
cd ..
cat > cluster.yaml <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: bottlerocket-cis-blog-eks
region: us-east-1
version: '1.24'
managedNodeGroups:
- name: bottlerocket-mng
instanceType: m5.large
desiredCapacity: 1
amiFamily: Bottlerocket
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
ssh:
allow: false
bottlerocket:
settings:
motd: "Hello from eksctl! - custom user data for Bottlerocket"
bootstrap-containers:
# 3.4
cis-bootstrap:
source: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$BOOTSTRAP_ECR_REPO:latest
mode: always
kernel:
# 1.5.2
lockdown: "integrity"
modules:
# 1.1.1.1
udf:
allowed: false
# 3.3.1
sctp:
allowed: false
sysctl:
# 3.1.1
"net.ipv4.conf.all.send_redirects": "0"
"net.ipv4.conf.default.send_redirects": "0"
# 3.2.2
"net.ipv4.conf.all.accept_redirects": "0"
"net.ipv4.conf.default.accept_redirects": "0"
"net.ipv6.conf.all.accept_redirects": "0"
"net.ipv6.conf.default.accept_redirects": "0"
# 3.2.3
"net.ipv4.conf.all.secure_redirects": "0"
"net.ipv4.conf.default.secure_redirects": "0"
# 3.2.4
"net.ipv4.conf.all.log_martians": "1"
"net.ipv4.conf.default.log_martians": "1"
EOF
Bottlerocket configuration settings are passed through to the managed nodes through user data in TOML format as referenced above to include the setting referencing the bootstrap container we created. To provision the cluster, run the following:
eksctl create cluster -f cluster.yaml
The cluster provisioning takes 10 to 15 minutes — now would be a great time to grab a cup of coffee.
Once the cluster is created, ensure that kubectl is functional:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-16-96.ec2.internal Ready <none> 4m36s v1.24.6-eks-4360b32
Once the cluster has been provisioned, you can verify the bootstrap container ran successfully on the Bottlerocket host. Since we configured our node group without SSH access, we’ll use AWS Systems Manager Session Manager to connect to the Bottlerocket control container, enter the admin container, and through the admin container obtain access to a host root shell:
aws ssm start-session --target $(aws ec2 describe-instances --filters "Name=tag:Name,Values=bottlerocket-cis-blog-eks-bottlerocket-mng-Node" | jq -r '.[][0]["Instances"][0]["InstanceId"]')
[ssm-user@control]$ enter-admin-container
[root@admin]# sudo sheltie
bash-5.1# journalctl -u bootstrap-containers@cis-bootstrap.service
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal host-ctr[1598]: time="2022-11-22T15:37:17Z" level=info msg="successfully started container task"
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal host-ctr[1598]: time="2022-11-22T15:37:17Z" level=info msg="container task exited" code=0
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal bootstrap-containers[1711]: 15:37:17 [INFO] bootstrap-containers started
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal bootstrap-containers[1711]: 15:37:17 [INFO] Mode for 'cis-bootstrap' is 'once'
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal bootstrap-containers[1711]: 15:37:17 [INFO] Turning off container 'cis-bootstrap'
Nov 22 15:37:17 ip-192-168-42-35.ec2.internal systemd[1]: Finished bootstrap container cis-bootstrap.
bash-5.1# exit
exit
[root@admin]# exit
exit
[ssm-user@control]$ exit
exit
Exiting session with sessionId: i-0d45e819f38a652ea-09be5b7991b5823d4.
Deploy a sample nginx pod on the Bottlerocket nodes
With a working cluster and managed node group, we can deploy a sample application to make sure everything is running properly. For this example, we’ll use a simple nginx deployment defined in the GitHub repository to deploy pods to the cluster. We can then verify the pods are running, and the nginx webserver started correctly:
cat > deploy-nginx.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOF
kubectl apply -f deploy-nginx.yaml
kubectl get pod
The output looks like below:
NAME READY STATUS RESTARTS AGE
nginx-74d589986c-xqkqb 1/1 Running 0 8s
Run the below command to exec into the pod and run a curl command:
POD_NAME=$(kubectl get pods -l=app=nginx -o=jsonpath={.items..metadata.name})
kubectl exec -it $POD_NAME -- /bin/bash
root@nginx-74d589986c-xqkqb:/# curl 127.0.0.1:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
....
root@nginx-74d589986c-xqkqb:/# exit
exit
Run the below command to access the logs from the pod and then Ctrl + C to exit from the container log output:
kubectl logs -f $POD_NAME
The output looks like the following:
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
...
...
2022/11/23 08:12:47 [notice] 1#1: start worker process 29
2022/11/23 08:12:47 [notice] 1#1: start worker process 30
127.0.0.1 - - [23/Nov/2022:08:24:29 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/7.74.0" "-"
^C
Validating the Bottlerocket AMI against the CIS Benchmark
After deploying a managed node group that adheres to the CIS Benchmark, we can use the commands outlined in the benchmark to verify the configuration, as 26 of the 28 checks are able to be automatically audited. For this step, we created a container to execute a script to perform the validation and a Kubernetes pod configuration to deploy the container to the cluster. This deployment method is idempotent in nature, so it can be executed once to verify initial configuration and regularly afterwards to detect any configuration drift.
First, we’ll need to again create an Amazon ECR repository to house the validation image:
cd bottlerocket-cis-validating-image
chmod +x create-ecr-repo.sh
./create-ecr-repo.sh
Second, we’ll build the validation container and put it in the newly created Amazon ECR reposition:
Third, we will use another cat command to insert the environment variables defined previously into the job-eks.yaml file located at the root of the GitHub repository. This file is used to deploy the Kubernetes batch job object which references the validation image onto the cluster:
cd ..
cat > job-eks.yaml <<EOF
---
apiVersion: batch/v1
kind: Job
metadata:
name: eks-cis-benchmark
spec:
ttlSecondsAfterFinished: 600
template:
metadata:
labels:
app: eks-cis-benchmark
spec:
hostNetwork: true
nodeSelector:
eks.amazonaws.com/nodegroup: bottlerocket-mng
containers:
- name: eks-cis-benchmark
image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$VALIDATION_ECR_REPO
imagePullPolicy: Always
securityContext:
capabilities:
add: ["SYS_ADMIN", "NET_ADMIN", "CAP_SYS_ADMIN"]
volumeMounts:
- mountPath: /.bottlerocket/rootfs
name: btl-root
volumes:
- name: btl-root
hostPath:
path: /
restartPolicy: Never
EOF
Apply the batch job using kubectl and check if the pod completed the execution:
kubectl apply -f job-eks.yaml
kubectl get Job,pod
The output looks like the following:
jp:~/environment/eks-cis-bottlerocket (main) $ kubectl get Job,pod
NAME COMPLETIONS DURATION AGE
job.batch/eks-cis-benchmark 1/1 5s 43s
NAME READY STATUS RESTARTS AGE
pod/eks-cis-benchmark-c4k5h 0/1 Completed 0 43s
pod/nginx-6c8b449b8f-29ldd 1/1 Running 0 5m20s 0 2m27s
Once the batch job has completed, we can view the pod logs to verify the CIS Bottlerocket Benchmark compliance status of the node:
POD_NAME=$(kubectl get pods -l=app=eks-cis-benchmark -o=jsonpath={.items..metadata.name})
kubectl logs $POD_NAME
The output should look like the following:
This tool validates the Amazon EKS optimized AMI against CIS Bottlerocket Benchmark v1.0.0
[PASS] 1.1.1.1 Ensure mounting of udf filesystems is disabled (Automated)
[PASS] 1.3.1 Ensure dm-verity is configured (Automated)
[PASS] 1.4.1 Ensure setuid programs do not create core dumps (Automated)
[PASS] 1.4.2 Ensure address space layout randomization (ASLR) is enabled (Automated)
[PASS] 1.4.3 Ensure unprivileged eBPF is disabled (Automated)
[PASS] 1.4.4 Ensure user namespaces are disabled (Automated)
[PASS] 1.5.1 Ensure SELinux is configured (Automated)
[PASS] 1.5.2 Ensure Lockdown is configured (Automated)
[PASS] 2.1.1.1 Ensure chrony is configured (Automated)
[PASS] 3.1.1 Ensure packet redirect sending is disabled (Automated)
[PASS] 3.2.1 Ensure source routed packets are not accepted (Automated)
[PASS] 3.2.2 Ensure ICMP redirects are not accepted (Automated)
[PASS] 3.2.3 Ensure secure ICMP redirects are not accepted (Automated)
[PASS] 3.2.4 Ensure suspicious packets are logged (Automated)
[PASS] 3.2.5 Ensure broadcast ICMP requests are ignored (Automated)
[PASS] 3.2.6 Ensure bogus ICMP responses are ignored (Automated)
[PASS] 3.2.7 Ensure TCP SYN Cookies is enabled (Automated)
[PASS] 3.3.1 Ensure SCTP is disabled (Automated)
[PASS] 3.4.1.1 Ensure IPv4 default deny firewall policy (Automated)
[PASS] 3.4.1.2 Ensure IPv4 loopback traffic is configured (Automated)
[PASS] 3.4.1.3 Ensure IPv4 outbound and established connections are configured (Manual)
[PASS] 3.4.2.1 Ensure IPv6 default deny firewall policy (Automated)
[PASS] 3.4.2.2 Ensure IPv6 loopback traffic is configured (Automated)
[PASS] 3.4.2.3 Ensure IPv6 outbound and established connections are configured (Manual)
[PASS] 4.1.1.1 Ensure journald is configured to write logs to persistent disk (Automated)
[PASS] 4.1.2 Ensure permissions on journal files are configured (Automated)
26/26 checks passed
Cleanup
Use these commands to delete the resources created during this post:
kubectl delete -f job-eks.yaml
kubectl delete -f deploy-nginx.yaml
eksctl delete cluster -f cluster.yaml --wait
aws ecr delete-repository --repository-name ${BOOTSTRAP_ECR_REPO} --force
aws ecr delete-repository --repository-name ${VALIDATION_ECR_REPO} --force
Conclusion
To comply with regulatory requirements, many organizations need the ability to deploy infrastructure that implements the recommendations from the Center for Internet Security Benchmarks. In this post, we showed you how an organization can bootstrap the Amazon EKS optimized Bottlerocket AMI and validate it against the CIS Bottlerocket Benchmark v1.0.0. This post used Amazon EKS Managed Node Groups to bootstrap Bottlerocket nodes; however, you can use the solution proposed above with self-managed node groups as well. After deployment, the validating container described above can be used. We hope this guide simplifies adopting Bottlerocket.
Check out our Containers Roadmap!
If you have ideas about how we can improve Amazon container services, then please use our Containers Roadmap to provide us feedback and review our existing roadmap items.