Enable traffic encryption between services in AWS App Mesh using AWS Certificate Manager or customer-provided certificates

NOTICE: October 04, 2024 – This post no longer reflects the best guidance for configuring a service mesh with Amazon ECS and Amazon EKS, and its examples no longer work as shown. For workloads running on Amazon ECS, please refer to newer content on Amazon ECS Service Connect, and for workloads running on Amazon EKS, please refer to Amazon VPC Lattice.

——–

Today, we announce the general availability of an AWS App Mesh feature that enables traffic encryption between services using AWS Certificate Manager (ACM) or customer-provided certificates. We sought feedback from our customers last year through the AWS App Mesh roadmap issues #38 and #39 and the features were made available on the AWS App Mesh Preview Channel for customers to try them.

Customers building mesh architectures may have security and compliance baselines where traffic between services is required to be encrypted. Additionally, enforcing the use of TLS, and validating the certificate from the upstream service, is an important aspect of a zero-trust network. Using this feature allows you to ensure that connectivity is secure and traffic is transmitted to a trusted service.

In App Mesh, traffic encryption works between Virtual Nodes, and thus between Envoys in the service mesh. This means that the application code is not responsible for negotiating a TLS-encrypted session, instead allowing the local proxy to negotiate and terminate TLS on application’s behalf.

Figure 1: Traffic encryption flow between services in the context of AWS infrastructure in use

Virtual Node Certificate Sources: ACM and customer-provided certificates

In GitHub roadmap issue #38 we added the ability to source certificate from a local filesystem, and in issue #39, we added the ability to source a Virtual Node’s certificate from ACM. When configuring a certificate for a Virtual Node, customers can choose one of these available sources.

Figure 2: AWS App Mesh traffic encryption using ACM-managed certificates

Some customers prefer ACM to manage their certificates. Using this option, ACM will automatically renew certificates that are nearing the end of their validity, and App Mesh will automatically distribute the renewed certificates. Using one of our walk-through illustrations, Figure 2 shows the traffic encryption between the gateway and the ColorTeller Virtual Nodes (white and green) using ACM.

Note: The above illustration in figure 2 can be tested using a sample walk-through available on GitHub. Link to examples are also available below within the ‘Hands-on walk-throughs and documentation’ section.

Virtual Node Backend Configuration: Client Policy and Client Policy Defaults

When we think about the relationship between connected services, we often discuss this relationship in the standard terms of the client (the entity sending traffic) and the server (the entity receiving that traffic). In App Mesh, we’re representing these concepts with new additions to the Virtual Node resource.

In App Mesh, a Virtual Node’s backends can be thought of as known destinations that the Virtual Node requires to communicate with. In this way, backends are the server of the client-server relationship. These backends are represented as Virtual Services in App Mesh, which are often owned by other teams or organizations. We make resilient connectivity between the client and server as easy as possible by allowing the Virtual Service owner to define settings that will automatically be distributed to clients, such as retry strategies and health checking configuration.

However, there’s often a need for the client to specify settings that the server cannot or should not provide. In the case of TLS, the client needs to indicate which certificate authorities they trust in order to validate that the server (backend) they are talking to is authentic and trusted.

To support the need to specify various settings from the perspective of the client, we’re adding a new concept called a ‘ClientPolicy’ to a Virtual Node’s backend. We also recognize that these settings will often be the same for many or all of a Virtual Node’s backend. To support this need, we’re additionally introducing the concept of ‘backendDefaults’ to Virtual Nodes, which allow customers to provide default settings that apply to all backends. Backend defaults can be overridden on a per-backend basis where necessary.

Today, the Client Policy configuration adds settings specific to TLS origination and validation, but will be enhanced in the future to support additional settings for clients, such as custom timeouts. For TLS-specific settings, customers are able to specify whether they want TLS enforced (i.e. a negotiation is attempted, and connection fails if it cannot), optionally what ports they want to enforce it on, and the certificate authorities to use when validating the certificate. For the certificate authorities to use, customers have two options to choose from:

A set of ACM Private Certificate Authorities.
A reference to the local file system where the collection of root certificate authorities (i.e. the trust bundle) is installed.

Enforcing Virtual Node Backend Client Policy with an ACM Certificate Authority ARN

The following example shows a client policy set on a Virtual Node’s backend to a Virtual Service, which enforces the use of TLS on port 443 only using ACM as the trusted Certificate Authority (CA). Note the use of ACM Private Certificate Authority (PCA) within the trust acm section.

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local",
                "clientPolicy": {
                    "tls": {
                        // (OPTIONAL) Whether or not TLS is enforced of this backend. 
                        // Default: true
                        "enforce": true,
                        // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                        // Default: all ports
                        "ports": [443],
                        // (REQUIRED) Validation context for TLS connections to this backend.
                        "validation": {
                            // (REQUIRED) Settings for determining where to retrieve the chain of trust.
                            "trust": {
                                "acm": {
                                    // (REQUIRED) The ARNs of trusted CAs.
                                    "certificateAuthorityArns": [
                                        "arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/17d7df97-2564-4d44-93a8-3aed69f5b294"
                                    ]
                                }
                            }
                        }
                    }
                }
            }
        }
    ]
}

Note: In the above example, both enforce and ports in the client policy's TLS settings are optional. If you don't specify them, TLS will be enforced on all ports by default.

You can specify up to 3 ACM certificate authority ARNs to use when validating the upstream certificate. If one of the certificate authorities has signed the upstream certificate, then it is considered valid.

Enforcing Virtual Node Backend Client Policy with customer-provided certificates

A common pattern for enforcing TLS is to use the default chain of trust on the local file system (or some other locally stored chain) to validate the upstream service's certificate. Most operating system ship with a bundle of root certificate authorities that can be used to validate public services.

The following example demonstrates a client policy, which enforces TLS on port 443 using the local file system to derive the certificate authorities that will be used to validate the upstream service's certificate:

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local",
                "clientPolicy": {
                    "tls": {
                        // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                        // Default: all ports
                        "ports": [443],
                        // (REQUIRED) Validation context for TLS connections to this backend.
                        "validation": {
                            // (REQUIRED) Settings for determining where to fetch the chain of trust from.
                            "trust": {
                                "file": {
                                    // (REQUIRED) The path to the certificate chain.
                                    "certificateChain": "/path/to/cert-chain.pem"
                                }
                            }
                        }
                    }
                }
            }
        }
    ]
}

Pricing

For using ACM-sourced certificates, customers pay a monthly fee for the operation of each AWS Certificate Manager Private Certificate Authority until they delete it. Customers also pay for the private certificate issued each month and exported private certificates. For more information, see AWS Certificate Manager Pricing.

Hands-on Walk-throughs and Documentation

To try the feature, customers can use the GitHub walk-throughs within App Mesh example repository to test both features linked below. These examples use the Color App that has been used previously for continuity and ease of explanation.

ACM-sourced certificates walk through: https://github.com/aws/aws-app-mesh-examples/tree/master/walkthroughs/tls-with-acm
Customer-provided certificates walk through: https://github.com/aws/aws-app-mesh-examples/tree/master/walkthroughs/howto-tls-file-provided

You will find further details about these features and more guidance in the Transport Layer Security (TLS) section of AWS App Mesh documentation. We welcome your feedback through our AWS App Mesh roadmap channel and we’re excited for you to use this feature to build secure and compliant mesh applications!

Containers