Containers
How to establish private connectivity for ECS Anywhere
Introduction
In 2014, AWS announced Amazon Elastic Container Service (Amazon ECS), a fully managed service that helps you orchestrate, deploy, and scale containerized applications. Although Amazon ECS serves a wide variety of customers from different segments, sizes, and verticals, there are cases where the applications need to run locally. For example, this often occurs in highly regulated industries. To resolve these challenges, we extended Amazon ECS and in 2020 announced Amazon ECS Anywhere, an option where customers can use the simplicity and functionality of Amazon ECS to run their container-based application outside an AWS Region, for example in their own premises, either using Virtual Machines (VMs), bare metal servers, or other customer-managed infrastructure.
ECS-A control plane runs on AWS, the Amazon ECS Application Programming Interfaces (APIs) are same, but your container-based applications can be hosted in any self-managed hardware, anywhere. This separation between control plane and data plane requires a communication path that’s between your corporate datacenter and Amazon ECS endpoints, and often a private communication channel is a requirement.
In this post, we explore how you can establish a private communication channel between your Amazon ECS Anywhere-managed container-based applications, hosted outside an AWS Region, and the correspondent Amazon ECS control plane APIs in a secure, and private manner using AWS Site-to-Site VPNs (S2S VPN) or AWS Direct Connect (DX).
ECS-A solution architecture
With Amazon ECS Anywhere, customers can schedule and run container workloads at scale, which are powered by Amazon ECS while using their own infrastructure. By registering their own VMs as Amazon ECS-managed instances. This requires connectivity between the customer’s premises and an AWS Region. which can happen either publicly through the internet, or privately with AWS Direct Connect or a Virtual Private Network (VPN).
As part of the configuration, the on-premises services require both an AWS Systems Manager agent as well as an Amazon ECS agent.
The SSM agent is a component of AWS Systems Manager that performs the on-host functions that are required by AWS Systems Manager. In this case, for the instance registration process, AWS Systems Manager generates and sends an activation key and instance role credentials upon request, which allows the execution of AWS APIs. The other component is the Amazon ECS agent, which communicates with the Amazon ECS control plane in an AWS Region, registers the server or VM as a managed instance, and receives and runs instructions like container configuration or task scheduling. Figure 1 shows this architecture.
Solution overview
ECS-A private communication architecture
Despite the native tolerance to disconnections and unreliable connectivity of Amazon ECS Anywhere, private connectivity between the AWS Region and the customer premises ensures a higher-level security, reliability, and performance. Figure 2 shows an architecture that allows the Amazon ECS and SSM agents to communicate privately and securely with their control plane APIs. This works by creating private VPC endpoints to ECS Anywhere control plane APIs, and forwarding the respective DNS queries to a Route 53 Inbound Resolver endpoint, that responds with private IP addresses. The agents communicate with the private addresses through a private communication channel (for example a DX or Site-to-Site VPN).
Prerequisites
This solution starts from a baseline of:
- An AWS account
- An Amazon ECS Anywhere cluster
- Private connectivity using AWS Direct Connect and/or Site-to-Site VPN between a VPC and your on-premises environment
Private communication channel
In this post, we aren’t going to describe in detail how you configure a private communication channel between your on-premises environment and an AWS VPC. However, there are two options: (1) AWS Direct Connect and (2) Site-to-Site VPN.
Regardless of your choice, when this communication channel is in place, you ‘ll have bidirectional connectivity using private IP addresses between your AWS and on-premises environment.
Walkthrough
You can deploy the following components in any order:
- Amazon ECS and AWS Systems Manager interface VPC endpoints (AWS PrivateLink) in an Amazon VPC
- Route 53 Resolver inbound endpoints in an Amazon VPC
- DNS conditional forwarding on your on-premises environment
Lastly, you configure Amazon ECS-A agent in your on-premises environment to communicate privately with AWS.
AWS PrivateLink
AWS PrivateLink provides private connectivity between your VPCs, AWS services and on-premises networks without exposing your traffic to the internet. To use AWS PrivateLink, you create an Interface VPC Endpoint. By creating a VPC Endpoint, a network interface is created in the subnet you specify and a private IP from your subnet’s address space is assigned to it. For this architecture, you’ll need to create six endpoints:
Optionally, if you are using Amazon Elastic Container Registry (Amazon ECR) to store your container images you’ll need the following additional endpoints, as described here:
And, if you want to receive logs in Amazon CloudWatch Logs you’ll also need the following endpoint:
Route 53 Inbound Resolver
You need to make sure that your on-premises resources resolve the DNS names of the endpoints to their private IP addresses. If you skip this step, then you’ll still resolve the public IP addresses for their domain names.
One option is to configure A records in your internal DNS server for the endpoints you just created. However, should you need to re-create the endpoints or change their subnets, you’ll have to manually change these records. In other words, you’ll be responsible to ensure that these records are kept updated.
The preferred solution is to create an Inbound Endpoint for an Amazon Route 53 Resolver. An Inbound Endpoint allows you resolve DNS queries in your VPC from external networks. You then configure a conditional forwarder so that your on-premises DNS server forwards queries of specific domains to the Route 53 Resolver. This way, you don’t have to worry about managing DNS records for resources inside your VPC.
Follow the steps in the Configuring inbound forwarding section for a step-by-step walkthrough. Figure 4 shows an example of a configured inbound endpoint with two IP addresses.
On-premises DNS resolution
The last step is to configure your on-premises DNS server to forward queries for the same six previously mentioned service.<region>.amazonaws.com domains to the IP addresses of your Route 53 Resolver inbound endpoints. This is highly dependent on your DNS server software. For the six endpoints in Figure 3, you’ll need to configure forwarding for the following DNS zones:
After these configurations, when your on-premises agents (Amazon ECS and SSM) try to reach AWS endpoints, they try to resolve the target domain name. The configured on-premises DNS server forward their DNS queries to the Route 53 Inbound Resolver and the resolver DNS responds with the private IP of the corresponding VPC endpoint. The agents communicate with the VPC endpoint private IP using the pre-established private communication channel (for example a DX or S2S VPN).
You can read more about Hybrid DNS Architectures in Hybrid Cloud DNS Options for Amazon VPC whitepaper.
To verify your DNS configuration and confirm you’re able to reach Route 53 Resolver endpoint’s IPs from your on-premises environment, you can use dig, nslookup, or any other command line utility that perform DNS lookups.
The result should output the private IP of your previously configured interface VPC endpoints, 10.0.0.45 in the previous example.
For configuration details and troubleshooting tips, the post How do I configure a Route 53 Resolver inbound endpoint to resolve DNS records in my private hosted zone from my remote network, provides a detailed configuration walkthrough of a Route 53 Resolver inbound endpoint.
SSM and ECS-A agents
At this point, you can setup Amazon ECS-A. Retrieve your clusters’ activation key, install both agents, and register the instance.
For detailed, step-by-step instructions, follow the Registering an external instance to a cluster documentation page.
Cleaning up
To avoid incurring costs, delete all endpoints and inbound endpoint you’ve created.
Delete VPC endpoints
- Open the VPC Console
- Navigate to Endpoints under Virtual Private Cloud
- Select the endpoints you created and click Actions -> Delete VPC endpoints
Delete Inbound endpoint
- Open the Route 53 Console
- Navigate to Inbound Endpoints in the Resolver section
- Select your inbound endpoint
- Select Delete
Conclusion
In this post, we showed you how to use private connectivity to achieve high levels of security and reliability with Amazon ECS Anywhere, with either AWS Site-to-Site VPN or AWS Direct Connect.
This post walked you through the configuration steps needed for the AWS side of the equation (i.e., interface endpoints and resolver endpoints) and high-level steps for the on-premises DNS forwarding.
Try this solution in your environment to make use of your private communications channels. You can also achieve similar traffic flows for other services like Amazon Simple Storage Service (Amazon S3) or Amazon CloudWatch. Learn more about it in Hybrid Networking using VPC Endpoints (AWS PrivateLink) and Amazon CloudWatch for Financial Services and Secure hybrid access to Amazon S3 using AWS PrivateLink posts.