Enable Private Access to the Amazon EKS Kubernetes API with AWS PrivateLink

Introduction

The adoption and large-scale growth of Kubernetes in recent years has resulted in businesses deploying multiple Amazon Elastic Kubernetes Service (Amazon EKS) clusters to support their growing number of microservice based applications. The Amazon EKS clusters are usually deployed in separate Amazon Virtual Private Clouds (Amazon VPCs) and often in separate AWS accounts.

A common challenge we hear from our customers is that their clients or applications have requirements to communicate with their Amazon EKS Kubernetes API across individual VPC and account-level boundaries and don’t want to route traffic over the Internet. In this post, we’ll explore a blueprint for using AWS PrivateLink as a solution to enable cross-VPC and cross-account private access for the Amazon EKS Kubernetes API.

Amazon VPC connectivity options

There are a number of AWS native networking services that offer connectivity across VPCs and across accounts without exposing your traffic to the public Internet. These include:

• VPC Peering – which creates a connection between two or more VPCs that allow endpoints to communicate as if they were on the same network. It can connect VPCs that are in different accounts and in different regions.
• AWS Transit Gateway – which acts as a highly scalable cloud router connecting multiple VPCs in a region with a hub-and spoke architecture. Transit Gateway works across accounts using AWS Resource Access Manager (AWS RAM).
• AWS PrivateLink – which provides private connectivity to a service in one VPC that can be consumed from a VPC in the same account or from another account. PrivateLink supports private access to customer application services, AWS Marketplace partner services, and to certain AWS services.

VPC Peering and Transit Gateway are great options for cross-VPC and cross-account connectivity when you want to integrate the VPCs into a larger virtual network. However, this may not be an option for some customers due to security concerns, compliance requirements, or overlapping CIDR blocks.

AWS PrivateLink

AWS PrivateLink is a purpose-built technology designed to enable private connectivity between VPCs and supported AWS services in a highly available, reliable, and secure manner. A service powered by PrivateLink is created directly inside your VPC as an interface VPC endpoint. For each subnet (from your VPC) you choose for your endpoint, an Elastic Network Interface (ENI) is provisioned with a private IP addresses from the subnet address range. This keeps your traffic on the AWS backbone network and removes the need for Internet and NAT gateways to access supported AWS services.

AWS PrivateLink offers several advantages when it comes to securely connecting services and resources across different VPCs and accounts:

1. PrivateLink provides a higher level of security and isolation by enabling private connectivity directly between VPCs without exposing traffic to the public Internet. This ensures that sensitive data and services remain within the AWS network, reducing potential attack vectors.
2. Setting up and managing PrivateLink is often simpler and more straightforward compared to VPC Peering and Transit Gateway, which can be complex to configure, especially in larger and more complex network architectures.
3. PrivateLink can be a preferred choice for organizations with strict compliance requirements, as it helps maintain data within the AWS network, which reduces the risk of data exposure.
4. PrivateLink can handle overlapping IP address ranges between VPCs. It simplifies network design in situations where address conflicts might occur.

AWS PrivateLink and Amazon EKS

Private access to the Amazon EKS management API is available with AWS PrivateLink. The EKS management API allows you to provision and manage the lifecycle of EKS clusters, EKS managed node groups, and EKS add-ons. Each EKS cluster you provision has a separate API (i.e., the Kubernetes API) that’s accessed using tools like kubectl and Helm to provision and manage the lifecycle of Kubernetes resources in that cluster.

An EKS cluster’s Kubernetes API is publicly available on the Internet by default. You can enable private access to the Kubernetes API by configuring your EKS cluster’s endpoint access. Historically, customers have enabled private access to clusters in other VPCs and accounts using VPC Peering or Transit Gateway. This blog introduces a way to use AWS PrivateLink instead to enable this private connectivity.

AWS PrivateLink for the Amazon EKS Kubernetes API

Native AWS PrivateLink support for the Amazon EKS Kubernetes API isn’t currently available, though there is an open roadmap request. However, customers can create a custom AWS PrivateLink powered service, known as a VPC endpoint service, for any application or service available behind a Network Load Balancer (NLB).

To create a PrivateLink powered endpoint service for the EKS Kubernetes API, you need to create a NLB that targets the Kubernetes control plane ENIs that EKS creates in your subnets. In addition, there are two challenges that need to addressed:

1. The EKS ENIs for the Kubernetes control plane are deleted and recreated in response to a number of events including, but not limited to: Kubernetes version upgrades, EKS platform upgrades, control plane autoscaling, enabling secret encryption, and associating/disassociating an OIDC identity provider. The NLB target group must be updated each time the ENIs are created and deleted.
2. A new DNS hostname is created for the PrivateLink interface VPC endpoint that isn’t included in the EKS Kubernetes API TLS certificate. Clients connections to the Kubernetes API using the PrivateLink endpoint DNS fail with a certificate error.

Solution overview

To addresses the challenge of the deleted and recreated Amazon EKS ENI’s behind the NLB target group, a combination of Amazon EventBridge and AWS Lambda are used to respond to these changes and automatically update the NLB target group. Two EventBridge rules are created to process these events with Lambda functions:

1. The first rule responds to the CreateNetworkInterface API call, which adds the newly created EKS ENIs to the NLB target group.
2. The second rule uses the EventBridge Scheduler to run every 15 minutes and removes targets for ENIs that have been deleted.

To address the challenge of the Kubernetes API certificate error, an Amazon Route 53 private hosted zone is created in the client VPC to override the DNS of the EKS API server endpoint. An alias record, matching the DNS name of EKS Kubernetes API server, is added to the private hosted zone and routes traffic to the AWS PrivateLink interface VPC endpoint in the client VPC. This eliminates certificate errors and enables secure and reliable communication with the EKS cluster.

This post introduces the Amazon EKS Blueprints PrivateLink Access pattern and the architecture is shown in the following diagram. PrivateLink is used to create a secure, private connection between the Client VPC and a private EKS cluster in the Service Provider VPC.

A few key points to highlight:

• The Amazon Elastic Compute Cloud (Amazon EC2) instance in the client VPC can access the private EKS cluster as if it were in its own VPC, without the use of an Internet gateway, NAT device, VPN connection, or AWS Direct Connect
• The client VPC has overlapping IPs with the Service Provider VPC. AWS PrivateLink makes it easy to publish an API or application endpoint between VPCs, including those that have overlapping IP address ranges.
• To connect through AWS PrivateLink, an interface VPC endpoint is created in the Client VPC, which establishes connections between the subnets in Client VPC and Amazon EKS in Service Provider VPC using network interfaces.
• This Service Provider creates a PrivateLink endpoint service pointing to internal NLB that targets the Kubernetes control plane ENIs.

Walkthrough

This post walks you through the following steps:

1. Download and configure the PrivateLink Access Blueprint
2. Deploy the PrivateLink Access Blueprint
3. Test access to EKS Kubernetes API server endpoint
4. End-to-end Testing with kubectl

Prerequisites

We need a few tools to deploy our blueprint. Ensure you have each of the following tools in your working environment:

• AWS Command Line Interface (AWS CLI) v2
• Session Manager plugin for AWS CLI
• Terraform
• kubectl
• jq

1. Download the PrivateLink Access Blueprint

In a terminal session, run the following set of commands to download the repo for the PrivateLink Access Blueprint:

git clone https://github.com/aws-ia/terraform-aws-eks-blueprints.git
cd terraform-aws-eks-blueprints/patterns/privatelink-access

2. Deploy the PrivateLink Access Blueprint

Run the following steps (in the order shown) to initialize the Terraform repository, create, and configure the Private NLB in the service provider VPC and Amazon EventBridge rules before the rest of the resources are created. This ordered execution of steps ensures that the NLB, Lambda functions and EventBridge rules are in place and ready to capture creation of (and/or changes to the) EKS Kubernetes API endpoints (i.e., ENIs) when the cluster is in the process of being created (or upgraded).

Note: The EKS Kubernetes API endpoint is temporarily made public so that Terraform has access to update the aws-auth ConfigMap. The Client Amazon EC2 instance’s AWS Identify and Access Management (AWS IAM) role is configured with administrative access for testing.

terraform init
terraform apply -target=module.eventbridge -target=module.nlb --auto-approve
terraform apply --auto-approve

Once the pattern has successfully deployed, you’ll be provided with multiple output values. Copy and save them to use later in the post. Review the Terraform output value for cluster_endpoint_private, it should look similar to snippet below:

aws eks update-cluster-config \
--region us-west-2 \
--name privatelink-access \
--resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

Copy the command and run it in a terminal session to take the cluster API endpoint private as originally intended. The above command causes the cluster to update, which takes a few minutes to complete. You can check on the status of the cluster by running the following command.

aws eks describe-cluster --name privatelink-access | jq -r '.cluster.status'

When the cluster is updating the output will be UPDATING , else the output will be ACTIVE.

3. Test access to EKS Kubernetes API server endpoint

Of the other Terraform output values, the value ssm_test is provided to aid in quickly testing the connectivity from the client EC2 instance to the private EKS cluster via PrivateLink. Copy the output value, which looks like the snippet shown below (as an example) and paste it into your terminal to execute and check the connectivity. If configured correctly, the value returned should be ok.

A successful test validates that:

1. Private hosted zone for EKS API server endpoint worked
2. The PrivateLink access worked thru the NLB all the way to the private API endpoints of the EKS cluster both of which are private in service provider VPC
3. The EKS Kubernetes API server endpoint is ready and functional

Note: The following snippet is shown only as an example.

COMMAND="curl -ks https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.gr7.us-west-2.eks.amazonaws.com/readyz"

COMMAND_ID=$(aws ssm send-command --region region-name \
   --document-name "AWS-RunShellScript" \
   --parameters "commands=[$COMMAND]" \
   --targets "Key=instanceids,Values=i-XXXXXXXXXXXXXXXXX" \
   --query 'Command.CommandId' \
   --output text)

aws ssm get-command-invocation --region region-name \
   --command-id $COMMAND_ID \
   --instance-id i-XXXXXXXXXXXXXXXXX \
   --query 'StandardOutputContent' \
   --output text

4. End-to-end testing with kubectl

On the Client EC2 instance, creating the kubeconfig file and interacting with the EKS cluster’s API server with kubectl should work seamlessly without any additional configuration as the IAM role associated with the Client EC2 instance is:

• added to the aws-auth ConfigMap and mapped to Kubernetes role of system:masters (cluster admin role)
• and based on the attached IAM policies, has the permissions to eks:DescribeClusterand eks:ListClusters

Follow the steps below to access the cluster’s API server with kubectl from the Client EC2 instance.

Log into the Client EC2 instance

Start a new AWS Systems Manager session (SSM session) on the Client EC2 instance using the provided ssm_start_session output value. It should look similar to the following code snippet shown (as an example). Copy the output value and paste it into your terminal to execute. Your terminal will now be connected to the Client EC2 instance.

Note: The following snippet is shown only as an example.

aws ssm start-session --region us-west-2 --target i-XXXXXXXXXXXXXXXXX

Update Kubeconfig

On the Client EC2 machine, run the following command to update the local ~/.kube/config file to enable access to the cluster:

aws eks update-kubeconfig --region us-west-2 --name privatelink-access

Test complete access with kubectl

Test access by listing the pods running on the cluster:

kubectl get pods -A

The output should look similar to the sample shown below:

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-4f8g8             1/1     Running   0          1m
kube-system   coredns-6ff9c46cd8-59sqp   1/1     Running   0          1m
kube-system   coredns-6ff9c46cd8-svnpb   1/1     Running   0          2m
kube-system   kube-proxy-mm2zc           1/1     Running   0          1m

Cleaning up

Before we destroy or teardown all the resources created, we need to ensure that the cluster state is restored for Terraform to do a complete cleanup. This means making the cluster API endpoint public again. Review the output value for cluster_endpoint_public, it should look similar to snippet below:

aws eks update-cluster-config \
--region us-west-2 \
--name privatelink-access \
--resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true

Copy the command and run it in a terminal session to take cluster API endpoint public and wait till the operation completes. Once verified that the cluster API endpoint is public, then proceed to clean up all the AWS resources created by Terraform by running the following command:

terraform destroy --auto-approve

Conclusion

In this post, we described an innovative blueprint for harnessing AWS PrivateLink as a solution to enable private, cross-VPC, and cross-account access to the Amazon EKS Kubernetes API. The solution involves PrivateLink, EventBridge, Lambda functions, and a Route53 Private Hosted Zone built as part of the EKS Blueprint to present an effective way to ensure private access to EKS Kubernetes APIs, which enhanced security and compliance for businesses leveraging microservices across multiple VPCs and accounts.

Reference

• De-mystifying cluster networking for Amazon EKS worker nodes