Zero operator access for self-managed databases using EC2 instance attestation

Self-managed databases often hold an organization’s most sensitive data ranging from private keys and credentials to personally identifiable information (PII) and financial transactions. Security sensitive applications subject to compliance and regulatory requirements cannot accept the risk of this data being accessed by those with access to the host and endeavor to implement zero operator access (ZOA) on the host.

In this post, you’ll learn how to design and implement a stateful database in a fully reproducible, ZOA environment powered by Amazon Elastic Compute Cloud (Amazon EC2), EC2 instance attestation, the Nitro Trusted Platform Module (NitroTPM), and Attestable AMIs. These service capabilities help bind the symmetric key of a Linux Unified Key Setup (LUKS) encrypted Amazon Elastic Block Store (Amazon EBS) volume to the cryptographic measurement of a specific Amazon Machine Image (AMI). The resulting EC2 instance can decrypt its storage only when launched with the correct Attestable AMI, whose expected boot measurements were calculated at the AMI build time.

While we demonstrate this technique with PostgreSQL as the example workload, the same pattern applies to any use case that requires strong guardrails for which workloads can access encrypted data, such as artificial intelligence (AI) model weights or proprietary algorithms.

Prerequisites and deployment

Before diving in, make sure you’re comfortable with the following:

• Nix Flakes: Nix Flakes is a fully self-contained build framework where a flake.nix file declaratively defines the entire system (kernel, packages, services, configuration) and a flake.lock file pins every input to exact git commit hashes, producing bit-for-bit reproducible AMI images whose content is fully determined by source code.
• Linux boot process: Unified Kernel Images (UKI), dm-verity for read-only filesystem integrity, and LUKS for encrypted volumes.
• TPM / PCR concepts: How Platform Configuration Registers (PCRs) accumulate boot measurements and why they can’t be forged.
• AWS services: EC2, KMS, EBS, and IMDS. You’ll need an AWS account with permissions to create KMS keys, EC2 instances, EBS volumes, and AWS Identity and Access Management (IAM) roles.

The complete reference implementation, including provisioning scripts and a NixOS-based image build, is available in the https://github.com/aws/nitrotpm-attestation-samples repository.

Choosing the right AWS confidential compute solution

AWS Nitro Enclaves is a popular confidential computing capability for stateless workloads. However, Nitro Enclaves operate in strict isolation with no persistent storage and restricted networking capabilities through the vsock channel. For a database workload that needs direct access to EBS volumes, high throughput and persistent state, building on enclaves would require significant engineering effort to work around these constraints.

EC2 instance attestation provides cryptographic proof that only trusted software and boot processes are running on an instance while preserving full hardware and network access. Combined with a reproducible build system like Nix Flakes, it delivers zero operator access without sacrificing the storage and performance characteristics databases demand.

For the ZOA database use case, EC2 instance attestation is the right choice. We get the confidentiality guarantees we need while retaining the ability to mount encrypted EBS volumes, serve high-throughput queries and operate with standard networking.

Construct a reproducible Attestable AMI

In this section we will dive deep into how Nix Flakes can be used to construct a fully reproducible and attestable AMI. Figure 1 lays out the 5 high-level steps required to build a reproducible and attestable AMI.

Figure 1: Amazon EC2 Instance Attestation — build time only, produce a reproducible AMI and generate its reference measurements.

1. Nix build: deterministic build from locked flake inputs produces the AMI raw image: ESP (UKI), dm-verity hash tree, and the nix store (erofs, read-only). Same inputs result in bit-identical AMI and thus results in the same PCR4 measurement.

2. dm-verity binds the store: the UKI cmdline carries the Merkle-tree root hash over the nix store. Any post-build change to the store alters the root hash which in turn alters PCR4. Optional, you can sign the UKI after the build to add a secure boot signature as done in the template using the --secure-boot flag.

3. nitro-tpm-pcr-compute: a build-time utility reads only the UKI .efi file and re-applies the measurement algorithm offline to calculate PCR values. PCR values represent a 96-character SHA-384 digest.

4. Reference measurements: the tool emits tpm_pcr.json:

• PCR4 (Boot Manager Code). Hashes the boot binaries the UEFI environment runs. With Attestable AMIs, this hash gets added to the Attestation Document and is evaluated by KMS for match with the configured PCR4 value in the KMS key policy. That way PCR4 pins exactly what code is authorized to interact with the KMS key.
• PCR7 (Secure Boot Policy, only with Secure Boot enabled). Hashes the UEFI Secure Boot policy. Because it persists across AMI updates, you can write one KMS policy that validates your Secure Boot certificate instead of a specific AMI measurement.
• PCR12. Hashes modifications applied to the kernel command line for example through addon files. Used with PCR4 to confirm the command line was not tampered with.

5. Bake into the AWS Key Management Service (AWS KMS)** key policy**: the reference measurements become RecipientAttestation conditions, so KMS will later release the key only to an instance whose actual measurements match.

Let’s now examine the implementation specifics of each step shown in Figure 1. Corresponding to steps 1 and 2, we first build a hardened Unified Kernel Image (UKI) that contains our Linux kernel, Kernel command line, initramfs and some metadata.

We recommend using Nix for the build, which makes the build declarative and reproducible. The same locked source consistently produces the same UKI bytes and thus always results in the same PCR4 measurement. The build emits a tpm_pcr.json file alongside the raw image as called out in steps 3 and 4 of Figure 1:

{
    "Measurements": {
        "PCR4": "f3a1e2...",
        "PCR7": "a32b31...",
        "PCR12": "000000..."
    }
}

To allow only an EC2 instance with Attestable AMI PCR measurements we control to run decrypt on an AWS Key Management Service (AWS KMS) key, we configure the AWS KMS key policy with the reference measurements.

{
    "Sid": "Allow decryption for the Instance role",
    "Effect": "Allow",
    "Principal": {
        "AWS": "${INSTANCE_ROLE}"
    },
    "Action": [
        "kms:Decrypt"
    ],
    "Resource": "*",
    "Condition": {
        "StringEqualsIgnoreCase": {
            ${PCR_VALUES}
        }
    }
}

${PCR_VALUES} of the KMS key policy json object contains measurement produced during the build tpm_pcr.json.

It is important to point out that the KMS key policy is the single enforcement boundary that releases decryption operations only to an AMI that has been cryptographically verified through instance attestation, which is why PCR values must be pinned and reviewed through a deliberate manual process rather than updated automatically by CI/CD pipelines. Automated updates could silently authorize a modified or unauthorized image and defeat the entire attestation guarantee.

In our example, we’ll create an AMI for a PostgreSQL database, take its PCR measurements, and use them to configure AWS KMS to only decrypt the database’s EBS volume key upon successful attestation with the right measurements. Let’s now look at a few specific aspects that Nix allows us to do in a descriptive and reproducible way.

Zero operator access

To enforce zero operator access and thus limit the surface area for unauthorized access, we disable any form of login to the machine, including ssh or serial access:

services.sshd.enable = lib.mkForce false;

and

systemd.services."autovt@"       = lib.mkForce {};
systemd.services."getty@"        = lib.mkForce {};
systemd.services.getty-static    = lib.mkForce {};
systemd.services."serial-getty@" = lib.mkForce {};

Firewall and Instance Metadata Service (IMDS) control

You can further limit inbound communication abilities to port 5432 required for PostgreSQL and grant outbound Instance Metadata Service (IMDS) access permissions to kms-init service only. IMDS access is required for services running on the EC2 instance to gain access to fresh temporary AWS credentials.

networking.firewall = {
    # Allow inbound PostgreSQL connections over mTLS
    allowedTCPPorts = [ 5432 ];

    # IMDS access control - only kms-init user can reach IMDS
    extraCommands = "
        # Allow kms-init service to access IMDS for KMS operations
        ${pkgs.iptables}/bin/iptables -A OUTPUT -d 169.254.169.254 -m owner --uid-owner kms-init -j ACCEPT
        [...]";
};

Measure the read-only filesystem

We have an ephemeral root (tmpfs, not backed by any disk or persistent storage) at /:

"/" = {
    fsType = "tmpfs";
    options = ["mode=0755"];
};

While the actual application data (for example, the postgres binary) in /dev/mapper/usr would be mounted read-only at /usr:

"/usr" = {
    device = "/dev/mapper/usr";
    # explicitly mount it read-only otherwise systemd-remount-fs will fail
    options = ["ro"];
    fsType = config.image.repart.partitions.${partitionIds.store}.repartConfig.Format;
};

It is important to note: the /dev/mapper/usr is not a part of the UKI and hence the NitroTPM measurements (the PCR values we calculate when building the UKI) are not directly tied to the data we stored in /dev/mapper/usr.

This is undesirable, because an unauthorized user who gains access to it could undermine the integrity and confidentiality of the database by, for example, swapping the postgres binary with a tampered build.

To verify the expected content of the mounted /dev/mapper/usr (mounted at /usr, containing the nix store) did not change, we tie it to the PCR measurements by using Linux dm-verity (as described in step 2 of Figure 1). When building the Attestable AMI, we calculate the Merkle Tree over /dev/mapper/usr and pass its root as a kernel boot parameter.

The kernel command line is embedded in the UKI at build time and covered by the PCR4 measurement of the whole binary. It includes systemd.verity_root_options=panic-on-corruption, so if dm-verity detects corruption on the /usr partition, the kernel panics and the instance reboots instead of running tampered code.

Secure boot variant

Some use cases require periodic rebuilds of the Attestable AMIs or allowing Attestable AMIs of different applications be subject to the same policy. Changes to the AMI reflect in the PCR4 measurements, and thus gating the AWS KMS policy on PCR4 values imply that every change to the AMI requires a corresponding change to the policy. If that is not desirable from an operational standpoint, it is possible to instead enable secure-boot and bind the policy to the resulting PCR7 measurement.

PCR7 is the measurement of the secure-boot state, which includes the identity of whoever signed the UKI. That means that, by enabling secure-boot during the Attestable AMI build and signing it with a predefined key, one could gate the access to the AWS KMS only on PCR7 measurement (dropping PCR4 and PCR12 from the policy), which would cover a whole fleet of Attestable AMIs and EC2 instances running them.

The boot-time trust chain

Let’s now have a look at how the trust chain is evaluated during boot-time. Figure 2 lays out the 8 steps of how the chain of trust is being established between NitroTPM, the attestable AMI and AWS KMS during EC2 boot.

Figure 2: Amazon EC2 Instance Attestation boot time trust chain. We use systemd services in the Attestable AMI which upon boot orchestrate the retrieval of the encrypted key, decryption, EBS volume unlock, mount, and database start.

0. UEFI measures the UKI: at boot AWS Nitro firmware loads the UKI from the ESP and extends NitroTPM: live UKI measured in PCR4, cmdline (all-zero) results in PCR12, Secure Boot keys (optional) results in PCR7.

1. Request attestation: network-online, an attestation document issued by NitroTPM (containing the actual PCR values) is provided to kms-init.

2. Fetch user-data: kms-init reads encrypted keys from Instance Metadata Service (IMDS) user-data.

3. KMS Decrypt: kms-init sends the signed attestation document + ciphertext to KMS. KMS compares actual vs. reference measurements. On match it returns the symmetric key (held only in tmpfs).

4. luks-unlock: unlocks the LUKS EBS volume using Linux cryptsetup to read and write to the encrypted volume with the symmetric key.

5. cert-init (parallel): decrypts the server cert bundle with the same key for PostgreSQL Transport Layer Security (TLS).

6. data.mount: mounts the unlocked ext4 volume from step 4 at /data, the only writeable volume that isn’t tmpfs and thus ephemeral.

7. postgresql: starts using TLS certs from cert-init, runs from the dm-verity protected nix store, reads/writes /data/postgresql.

Extending the same key to a TLS server identity during boot-time

Let’s now examine how confidential information can be managed during boot-time using the KMS protected symmetric key.

The symmetric key that the boot chain releases is generic. Any artifact encrypted with this key at provisioning time can be decrypted in the same attested boot. The reference implementation uses this property to provision a TLS server identity for PostgreSQL that is also bound to the AMI’s measurement. The certificate authority (CA), the server certificate, and the server private key are packaged into a tarball, encrypted with the same symmetric key, and embedded in the EC2 user data alongside the database ciphertext. The cert-init.service decrypts the bundle inside the attestable AMI into a tmpfs directory, and PostgreSQL serves on port 5432 with clientcert=verify-full enforced, establishing mutual TLS with clients that hold certificates signed by the same CA.

Build and deploy the reference implementation

The reference implementation is ready to run end-to-end. Consult the Readme to learn about all different deployment options such as --secure-boot or --debug.

1. Clone the repo and change into it:

git clone https://github.com/aws/nitrotpm-attestation-samples && cd nitrotpm-attestation-samples

2. Verify sufficient IAM permissions as called out in Minimal IAM Privileges section in Readme.
3. Run the following commands from the repository to provision the AWS resources and launch an EC2 instance with PostgreSQL running on an attested LUKS-encrypted volume using secure boot and secrets-manager:

cd nix/examples/postgres-kms
./scripts/e2e-test.sh --secure-boot --secrets-manager --authorize-my-ip --create-roles --no-cleanup

Clean up

To remove all AWS resources created by the example, run:

./scripts/clean.sh

Conclusion

In this post, we described the reproducible build process of a PostgreSQL Attestable AMI with zero operator access using Amazon EC2 instance attestation. We took a deep dive into the NitroTPM based boot validation process and how encrypted EBS volumes can be mounted after successful boot. We discussed how changes to the AMI can be detected using PCR measurements to prevent the EC2 instance from booting and preventing tampered code from gaining sufficient KMS decrypt permissions.

Now go, clone the nitrotpm-attestation-samples repo, customize the template for your own needs and start securing your data with Attestable AMIs!

About the authors