Unlocking Benefits with Bottlerocket: A Purpose-Built Container OS

Post updated on Nov 26, 2024.

This post discusses Bottlerocket, a Linux-based open source operating system that is purpose-built for running containers. We will walk through fleet management at scale and how users can benefit from it. We will also share the key pillars that support Bottlerocket’s development plan.

The challenges of fleet management with General-Purpose host Operating Systems

General-Purpose Operating Systems (GPOS) are designed to accommodate a wide range of applications and use cases, providing a versatile environment for diverse computing needs. However, this inclusivity presents challenges in terms of resource optimization and fleet management. In environments with a large GPOS footprint, the multitude of software packages often leads to version disparities across systems, making fleet management complex. Coordinating updates and ensuring uniformity becomes a formidable task, especially in large-scale deployments where compatibility must be preserved for a wide variety of applications.

To reduce some of the compatibility challenges, GPOS vendors typically offer long-term support (LTS): a product lifecycle management policy not to make incompatible changes to software packages, combined with a promise to provide security and bug fixes. It is difficult to keep the compatibility promise while continuously updating open source software to the latest versions, and expensive to keep the support promise by back-porting patches and fixes to older versions. These challenges and costs grow over time, which leads to typical LTS lifetimes of five to ten years for a GPOS, with the user migrating to a newer version before the support period ends.

Motivation for Single-Purpose Operating Systems

Single-Purpose Operating Systems (SPOS) are designed and optimized for specific tasks. They are common in embedded environments or devices with a constrained user interface or API, such as a wireless router. They may not support remote login or interactive shell access except through specialized debugging mechanisms. They are often image-based rather than package-based. On package-based systems, software is decomposed into packages that can be installed and updated independently by a package manager. On image-based systems, software is pre-installed into a filesystem image and must be installed and updated all at once. This uniformity minimizes the potential delta between the state of the system during pre-release testing and the actual end state after applying an update on production systems, easing a frequent source of user pain.

For an SPOS, the software it ships is an implementation detail that users cannot directly depend on. It is still bounded by Hyrum’s Law – all observable behaviors of the system will be depended upon by someone, but the operating system can restrict access to the specific library versions and command-line programs installed, and mandate the use of the API instead. This means that the majority of software can be updated to the latest versions without breaking any compatibility promise to users, which minimizes the expense of ongoing support. Consequently, an SPOS might not define an LTS lifetime at all.

Single-Purpose Operating System for Container Hosts

Containers are a popular solution to the fleet management problem in GPOS environments. Each host in the fleet can run multiple containers, and each container, while sharing the same kernel, can include a different set of packages with different configurations depending on the needs of the application. The trade-off is that it is no longer possible to update a package on the host and have that update affect all running applications; the benefit is that fleet-wide package updates no longer need to move at the pace of the slowest-moving application. In addition, once all the applications run inside containers, the packages installed on the host no longer matter, which means the host may not need a package manager at all. This freedom makes it possible to use an SPOS to further simplify fleet management when running container workloads.

Moving to image-based deployments eliminates the problem of diversity at the host level, which in turn simplifies fleet management by:

• Minimizing the potential for untested combinations of software, and
• Eliminating the risk of unexpected local edits. If the SPOS does not define an LTS and does not require migrations, then fleet management can be further simplified.

Introduction to Bottlerocket

Bottlerocket is an open source, container-optimized Linux distribution – an SPOS designed for the exclusive purpose of hosting containers. Each Bottlerocket image, known as variant, is specialized for a specific combination of container orchestrator and cloud environment. Bottlerocket variants are provided to users as a set of Amazon Machine Images (AMI), currently optimized for either Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS), with an associated stream of images for in-place updates. These AMIs are available for use at no additional cost, and are covered by AWS support plans. The end-of-life for a given variant is tied to a software lifecycle that falls under shared responsibility of users, such as minor versions of Kubernetes or newer versions of the Linux kernel. Otherwise, at an operating system level, Bottlerocket does not have specific end-of-life policy.

Bottlerocket images do not have an SSH server or even a shell. Instead, they are API-driven and offer an API to adjust system settings and to trigger actions such as a reboot. Containers are the only supported mechanism for running custom code on Bottlerocket. Bottlerocket facilitates systems management tasks by providing two optional add-on containers: one for AWS Systems Manager (SSM) based management, which is enabled by default and another for Secure Shell (SSH) access, which is disabled by default.

Reasons users choose Bottlerocket

Users experience benefits such as increased uptime for container applications, improved security and resource utilization, lower management overhead and operational cost, and faster node ready times. This is reflected in the customer and partner testimonials on the Bottlerocket Getting Started page. We believe that these benefits can be attributed to core Bottlerocket values of minimalism, operational reliability and out of the box compliance.

Security benefits from curated dependencies: The Bottlerocket team maintains around 100 package definitions, compared to 50,000 in a large GPOS. Having full control over each package definition ensures that unnecessary features and dependencies are disabled at build time. With a few exceptions, such as the kernel, all software can be updated to the latest upstream version at any time, since none of it is directly exposed to users. These attributes make it easier to avoid vulnerabilities in the first place, and to fix the ones that are identified.

Operational reliability through image-based updates: Bottlerocket offers in-place updates where both upgrade and rollback are fully tested before reaching users. This is a core product value that’s reflected in the supporting infrastructure: AMIs for every supported version of Bottlerocket can be discovered through SSM parameters, and any running Bottlerocket instance can be upgraded or downgraded to any version.

Compliance, out of the box: A golden AMI is an AMI that you standardize through configuration, consistent security patching, and hardening. It also contains agents you approve for logging, security, performance monitoring, etc. One of the most common reasons for creating a golden AMI is to harden it according to an industry benchmark like those published by the Center for Internet Security (CIS). Bottlerocket offers the only first-party AMIs that are hardened to CIS Bottlerocket Benchmark v1.0.0 optimized for use with Amazon ECS and Amazon EKS, and are fully supported by AWS. It doesn’t support interactive login via SSH or console, omits any software not required for running containers or managing the OS and enforces cryptographic integrity checks for the root filesystem and mandatory access controls such as SELinux to reduce the attack surface in the event of container escape, even if the compromised application is running as root. Bottlerocket also provides a built-in reporting tool and a Report API so that users can automate the collection of CIS benchmark results.

What if you want to customize the Bottlerocket variant?

Bottlerocket team builds and maintains a few variants but many Bottlerocket users are interested in creating their own custom Bottlerocket variants. Until recently, any additional variants could only be created by forking the Bottlerocket repository and maintaining a series of patches which adds the special variant. There are some risks and challenges associate with this method such as:

• The custom variant repository must frequently re-base a substantial volume of code to stay atop Bottlerocket changes. These re-bases are unpredictable and likely non-trivial in many cases.
• The integration points of user-defined custom variants, also known as out-of-tree variants are not well-defined. For example, changes to the semantic meaning of settings could impact out-of-tree variants in ways that are difficult to discern during re-base.
• The forked repository will include a significant amount of code/documentation/information which is not relevant to the specific changes needed for the custom variant.

We have recently solved the aforementioned challenges with our Bottlerocket Out of Tree Builds (OOTB) support, which is a mechanism for creating custom Bottlerocket variants that live within their own git trees. These variants define a dependency on Bottlerocket’s core in a way that allows the custom variant to evolve independently and consume Bottlerocket core updates safely. We have already migrated existing Bottlerocket variants to OOTB infrastructure. If you built a custom variant of Bottlerocket, you can now use OOTB to build it without forking and maintaining the entire Bottlerocket core repository. Please refer to design document here and we look forward to your feedback via Github Issue in this improved approach to build and maintain custom Bottlerocket variants.

Development Plan

The Bottlerocket development plan is primarily shaped by direct feedback from our users and is based on the following four pillars:

1. Supporting Bottlerocket across Amazon Elastic Compute Cloud (EC2) instance types such as Inf1, Inf2, Trn1 and AWS Regions
2. Doubling down on built-in compliance capabilities
3. Making Bottlerocket ramp-up process easier for users, and
4. Continuing to improve node ready times

While the first pillar is self-explanatory, for the second, we announced new AMIs for Bottlerocket that are preconfigured to use FIPS 140-3 validated cryptographic modules, including the Amazon Linux 2023 Kernel Crypto API and AWS-LC. With these AMIs, the host software uses FIPS-approved cryptographic algorithms for TLS connections. Bottlerocket is an API-driven operating system and we updated the Bottlerocket reporting API to include mechanisms that allow customers to validate that FIPS mode is enabled.

For the third pillar, we will simplify the user experience by making boot-time tasks easier to automate. Bottlerocket allows bootstrap containers as a mechanism to perform boot-time tasks such as configuring ephemeral storage, but today users have to bring their own container image and keep it up to date. We plan to create a generic bootstrap container image that executes scripts passed as user data, and can be used by users to configure their instances. This will remove the undifferentiated heavy-lifting for users to maintain their own customized solutions for common use cases such as using ephemeral storage to store container images. In addition, we now support bootstrap commands that enable users to run predefined commands such as configuring node-local NVM Express (NVMe) storage through the Bottlerocket API without taking any runtime dependency on bootstrap containers.

For the final pillar, we will continue to improve node launch times. A recent example of this is an optimization from Bottlerocket release 1.21.0 to ship the “pause” container as part of the OS build. This is a very small container image that must be present on the node before any pod can be created. While the image pull time for the pause container is minimal, having it already present as a part of OS build makes node ready time even faster. Beyond that, we will work on bringing up the network and local storage in parallel to further speed up node launches.

Bottlerocket is an open source project and we welcome issues with new ideas, bug reports, and pull requests. We’d also love feedback from you about your experiences using Bottlerocket.

Get Started with Bottlerocket

Bottlerocket has a website that serves as a central hub for Bottlerocket related information, tutorials, and other resources to help users get started. If you are new to Bottlerocket, refer to the Get Started section on the website to learn more about using Bottlerocket as a container host OS.