Traffic Encryption in AWS App Mesh across accounts using certificates from AWS Certificate Manager Private CA and AWS Resource Manager

NOTICE: October 04, 2024 – This post no longer reflects the best guidance for configuring a service mesh with Amazon ECS and Amazon EKS, and its examples no longer work as shown. For workloads running on Amazon ECS, please refer to newer content on Amazon ECS Service Connect, and for workloads running on Amazon EKS, please refer to Amazon VPC Lattice.

Introduction

AWS App Mesh is a service mesh that provides application-level networking to standardize how your services communicate, giving you end-to-end visibility and enabling controls to tune for high-availability of your applications. Customers building mesh architectures and enforcing the use of TLS to validate the certificate from the upstream service, is an important aspect of a zero-trust network. Using this feature, traffic is guaranteed to be encrypted before it ever leaves your compute and you’re able to validate that the upstream destination is trusted to be who they say they are. App Mesh provides an integration with AWS Certificate Manager (ACM) Private Certificate Authority (PCA), to enable TLS between connected services in a manner that does not require changes to application code.

Customers often have applications owned by different accounts in different Amazon ECS clusters. App Mesh makes it easy to connect all services into a common mesh and enable configuration of encryption via the mesh. With the support of the Cross-Account feature of AWS Certificate Manager (ACM) Private Certificate Authority (CA), customers can create a CA in one account and use AWS Resource Access Manager (RAM) to share their CAs across the enterprise. This helps manage the overall cost of CA infrastructure, when the same organization is deploying services using separate AWS accounts. You can find more details about this feature here.

Benefits

Complete managed renewal capability
Each account the Private CA is shared with that creates a private certificate in its own account, remains in full control of the certificate because the key lives in the certificate creation account and is fully protected there. The certificate creation account can associate the certificate to any resource in their account or export it for further use. Each certificate that is created also has the complete managed renewal capability of ACM. By issuing private certificates via ACM, customer certificates are automatically renewed, preventing certificate based outages.
Cost Savings
Private CA cross-account sharing, gives you the ability to grant permissions for other accounts to use a centralized CA to generate and issue certificates by using AWS Resource Access Manager (RAM) to manage the permissions. In a configuration with a CA in each account, customers must pay a monthly fee for each CA. Using the shared CA model, customers only pay for the one CA that they are sharing, regardless of the number of to which accounts they share.

Overview

In this tutorial, you will create AWS App Mesh components and deploy them using a sample application called Yelb. Yelb allows users to vote on a set of alternatives like restaurants and dynamically updates pie charts based on the votes. Additionally, Yelb keeps track of the number of page views and prints the hostname of the yelb-appserver instance serving the API request upon a vote or a page refresh. Yelb components include:

A frontend called yelb-ui is responsible for vending the JS code to the browser.
An application server named yelb-appserver, a Sinatra application that reads and writes to a cache server (redis-server), and a PostgreSQL backend database (yelb-db).
Redis stores the number of page views and PostgreSQL stores the votes.

The following picture depicts the architecture.

Steps to resource share ACM – Private CA across accounts:

Create a Private CA that you will be sharing across the accounts. If you already want to use an existing CA, identify which Private CA(s) you want to share, and which accounts you want to share with.
Create a resource share and then add your ACM Private CA to the share.
Share the resource with a single account or with your AWS Organizations structure.
In the shared account(s), create a certificate through the ACM console, Share your Private CA with your AWS Organizations accounts and share your Private CA with individual accounts.

Deployment

Step 1: Set up the infrastructure

To follow this step by step implementation, you will need to have an environment with some tooling. An AWS Cloud9 instance can be used to run this tutorial. If you want to create a Cloud9 instance in your account, follow the steps in the EKS Workshop from the chapter Create a Workspace to Update IAM Settings for your Workspace.
These are the tools we will need to have installed:

The latest version of the AWS CLI
Docker
Git
jq
Helm
eksctl
kubectl

Since we will be running this in two different accounts, you need to make sure that you have profiles with credentials for both AWS accounts configured in ~/.aws/credentials and ~/.aws/config files. Here’s how it is going to look once you complete the profile setup successfully:

cat ~/.aws/credentials
[frontend]
aws_access_key_id = ...
aws_secret_access_key = ...
[backend]
aws_access_key_id = ...
aws_secret_access_key = ...

cat ~/.aws/config
[profile frontend]
region = us-west-2
[profile backend]
region = us-west-2

?You can get more information about creating configuration files in the AWS CLI user guide.

After installing everything and configuring the profiles, let’s clone the GitHub repository that has the scripts that we will be using in this tutorial:

git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/eks-app-mesh-cross-account-acm/

We will be using the CloudFormation templates to create the VPC, Security Groups, Route Tables, and the VPC Peering connection between the two accounts, if you want to take a look at them, they’re located in the infrastructure directory of the repo you just cloned. You can create them by running the following script:

./infrastructure/setup.sh

Step 2: Create the EKS clusters

In this step, we will use eksctl to create the EKS clusters in the frontend and backend accounts by running the following script:

./eks/setup.sh

You can see your new clusters by running kubectl.

kubectl config get-contexts

Step 3: Install the App Mesh components

Before creating the mesh resources and deploying the Yelb app onto the clusters, you need to install the AWS App Mesh Controller. This controller allows you to configure App Mesh resources using kubectl. You will use Helm to install the App Mesh controller.

First, install the controller in the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io
helm repo add eks https://aws.github.io/eks-charts
kubectl create ns appmesh-system
helm upgrade -i appmesh-controller eks/appmesh-controller \
--namespace appmesh-system

Now, install the controller in the Backend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io
kubectl create ns appmesh-system
helm upgrade -i appmesh-controller eks/appmesh-controller \
--namespace appmesh-system

Confirm that the App Mesh controller is running:

kubectl --context=<iam_user>@am-multi-account-1.<region>.eksctl.io get pods -n appmesh-system
kubectl --context=<iam_user>@am-multi-account-2.<region>.eksctl.io get pods -n appmesh-system

You should see outputs similar to this one:

NAME                                READY STATUS   RESTARTS AGE 
appmesh-controller-66b749c78b-67n68 1/1   Running  0        6s

Step 4: Deploy and share the mesh

Now that you have the App Mesh controller set up in both clusters, you want to create the mesh and share it to the Backend Account.

First, let’s create the namespace where you will deploy the mesh components and the Yelb application in the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io
kubectl create ns yelb

You need to add two labels to the yelb namespace: mesh and appmesh.k8s.aws/sidecarInjectorWebhook. These labels instruct the controller to inject and configure the Envoy proxies in the pods:

kubectl label namespace yelb mesh=am-multi-account-mesh
kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled

Let’s create and share the mesh and share it to the Backend Account:

./mesh/create_mesh.sh

Share the mesh using AWS Resource Access Manager:

aws --profile frontend cloudformation deploy \
--template-file shared_resources/shared_mesh.yaml \
--parameter-overrides \"BackendAccountId=$(aws --profile backend sts get-caller-identity | jq -r .Account)" \
--stack-name am-multi-account-shared-mesh \
--capabilities CAPABILITY_IAM

Now, accept the Resource Share Invitation in the Backend Account.

Run the following commands to accept the invitation:

RESOURCE_SHARE_ARN=$(aws --profile backend ram get-resource-share-invitations \| jq -r '.resourceShareInvitations[] | select(.resourceShareName=="mesh-share") | .resourceShareInvitationArn')

aws --profile backend ram accept-resource-share-invitation \
--resource-share-invitation-arn $RESOURCE_SHARE_ARN

The Backend Account needs a service role so App Mesh can discover yelb-appserver Virtual Node’s ip address using AWS Cloud Map. Create it with this command:

aws --profile backend iam create-service-linked-role \
--aws-service-name appmesh.amazonaws.com

Step 5: Deploy the mesh resources and the application

Since the resources in the Backend Account are dependencies for the resources in the Frontend Account, let’s start by creating them:

Switch to the Backend Account’s cluster context:

kubectl config use-context <iam_user>@am-multi-account-2.<region>.eksctl.io

Even though the mesh is already created and shared, the App Mesh controller in the Backend Account’s cluster needs to be aware of it, let’s run the following script:

kubectl create ns yelb
kubectl label namespace yelb mesh=am-multi-account-mesh
kubectl label namespace yelb "appmesh.k8s.aws/sidecarInjectorWebhook"=enabled
./mesh/create_mesh.sh

Now, create the Mesh components in the Backend Account, this will create Virtual Nodes, Virtual Services, and Virtual Routes for yelb-appserver, yelb-db and yelb-redis.

kubectl apply -f mesh/yelb-redis.yaml
kubectl apply -f mesh/yelb-db.yaml
kubectl apply -f mesh/yelb-appserver.yaml

Deploy yelb-appserver, yelb-db and yelb-redis.

kubectl apply -f yelb/resources_backend.yaml

Get the yelb-appserver Virtual Service ARN and change mesh/yelb-ui.yaml backend configuration accordingly. This is a manual process, yelb-ui’s Virtual Node creation will fail if skipped.

kubectl -n yelb get virtualservice yelb-appserver
# Replace <yelb-appserver virtualServiceARN># in mesh/yelb-ui.yaml with the ARN from the # command above using your editor of choice

Now that you have all the Backend Account components deployed, switch to the Frontend Account’s cluster.

kubectl config use-context <iam_user>@am-multi-account-1.<region>.eksctl.io

Create the mesh components in the Frontend Account, this will create a Virtual Node, Virtual Service, and a Virtual Route for yelb-ui.

kubectl apply -f mesh/yelb-ui.yaml

Deploy yelb-ui.

kubectl apply -f yelb/resources_frontend.yaml

Get the the Load Balancer URL to test the application in your browser:

kubectl get service yelb-ui -n yelb

Step 6: Configuring TLS Encryption on AWS App Mesh Virtual Nodes

In App Mesh, traffic encryption works between Virtual Nodes, and thus between Envoys in the service mesh. This means that the application code is not responsible for negotiating a TLS-encrypted session, instead allowing the local proxy to negotiate and terminate TLS on application’s behalf. In the below script, CA is created and shared with the backend account.

./mesh/apply_tls_on_yelb_nodes.sh

Enforcing Virtual Node Backend Client Policy with an ACM Certificate Authority ARN
Note: The below step is not required, apply_tls_on_yelb_nodes.sh above takes care of the required TLS encryption configurations. The screenshot is to show how it works when you are configuring via the AWS console.

The following screenshot shows a client policy set on a Virtual Node’s backend to a Virtual Service, which enforces the use of TLS on port 443 only using ACM as the trusted Certificate Authority (CA). Note the use of ACM Private Certificate Authority (PCA) within the trust ACM section.

You should see the application as follows, go ahead and test it!

Step 7: Cleanup

In order to clean up all the resources created during the execution of this tutorial, run the cleanup script with the following command:

./cleanup.sh

Note that this cleanup script won’t delete any AWS accounts created for this tutorial, but only the resources provisioned in it by the previous steps.

Conclusion

With this step by step tutorial, customers are encouraged to test this feature out and implement secure traffic between your services thus meeting your security and compliance baselines. You will find further details about these features and more guidance in the Transport Layer Security (TLS) section of AWS App Mesh documentation. We welcome your feedback through our AWS App Mesh roadmap channel and we’re excited for you to use this feature to build secure and compliant mesh applications!

Containers