Networking & Content Delivery

Migrate Amazon ECS service communication to Amazon VPC Lattice

Advanced L7 routing, authentication and authorization using IAM Policies, and deep observability are key capabilities offered by Amazon VPC Lattice. With VPC Lattice now integrated into Amazon Elastic Container Service (Amazon ECS), you can easily migrate service-to-service communication from using internal Elastic Load Balancing (ELB) to VPC Lattice. This allows you to benefit from VPC Lattice capabilities in your ECS deployments, without requiring you to maintain additional load balancing infrastructure.

Prerequisites

In this post I focus on migration best practices. To get a first overview of the VPC Lattice and ECS native integration, I recommend reading the post Streamline container application networking with native Amazon ECS support in Amazon VPC Lattice written by my colleague Donnie. I do not dive deep into VPC Lattice components, so I recommend reviewing the post Build secure multi-account multi-VPC connectivity for your applications with VPC Lattice.

Baseline architecture

In this post I use ECS-based applications fronted by Application Load Balancers (ALBs) for client-to-service and service-to-service communication. Figure 1 shows the baseline architecture. The client side is a web front-end service exposed externally through an internet-facing ALB. On the server side I configured two back-end ECS services: “cart” and “catalog”. The internal ALB uses host header base rules to route the cart.int.example.com and catalog.int.example.com Fully Qualified Domain Names (FQDN) to their respective ECS services. I configured Route 53 DNS records for the FQDNs in a Private Hosted Zone, using Alias A and AAAA records pointing to the internal ALB.

Figure 1: Baseline architecture

Figure 1: Baseline architecture

Overview of migration steps

First, I create VPC Lattice target groups for the backend ECS services, and configure ECS to dynamically add container IPs to the VPC Lattice target groups. Then I create the necessary VPC Lattice constructs, including the service network, services, listeners, health checks, service network associations, and map the target groups to the listeners.

Figure 2 shows the architecture after performing these steps. VPC Lattice is now ready to be used by the front-end service to route its requests to the back-end services. However, as the Route 53 Alias Records for the backend FQDNs still point to the ALB, traffic continues to flow through the internal ALB.

Figure 2: VPC Lattice, Prepared for Migration

Figure 2: VPC Lattice, Prepared for Migration

Before moving the traffic from ALB to VPC Lattice I perform the following validation steps:

  1. Check that the VPC Lattice target group shows all expected container IPs as targets.
  2. Ensure that all health checks are configured correctly on the new VPC Lattice target group, and all target show up as healthy.
  3. (Optional) Deploy a temporary test front-end service and target the VPC Lattice Services auto-generated FQDNs while rewriting the HTTP Host header to match the back-end service FQDNs. This allows you to verify that traffic is reaching your destinations before switching over the Route 53 record for your backends.

After successfully validating that VPC Lattice routes traffic from the front-end service to the back-end services, I am ready to move the production traffic over. For this, I change the Route 53 records for cart.int.example.com and catalog.int.example.com in the Route 53 Private Hosted Zone, and let them point to the VPC Lattice services FQDNs. Figure 3 shows the deployment architecture after moving traffic to VPC Lattice.

Figure 3: Route 53 Record switchover to VPC Lattice

Figure 3: Route 53 Record switchover to VPC Lattice

After performing the DNS change, there are a few factors that influence how fast client traffic shifts over to VPC Lattice:

  • Alias records in Route 53 have a TTL of 60 seconds. Only after the Route 53 synchronizes the change internally, and the TTL expires in the front-end client cache, new connections to the back-end service will use VPC Lattice. If the original Route 53 record use was a CNAME record, its TTL could be much higher. It therefore makes sense that you lower the TTL of the original CNAME record before doing the switchover.
  • The front-end service may use persistent HTTP sessions to communicate with the back-end services. These persistent sessions will continue to use the ALB until they expire, because the client will not make a DNS request until a new TCP session to the back-end services is created.

I recommend maintaining the ALB in place until you validate all communication flows. If you need, you can easily roll-back by reverting the DNS changes in the private hosted zone.

The final step is to delete the unused ALB constructs like Target Groups, Listeners, and the ALB itself. Before doing so, check the ActiveConnectionCount metric for the ALB and VPC Lattice Services in CloudWatch. If the metric stays persistently at 0 on the ALB and is greater than 0 on VPC Lattice, you can safely delete the ALB.

Detailed Migration Walkthrough

Prerequisites

  • Run the commands in the walkthrough in any bash or equivalent shell that has the AWS CLI installed, or use the AWS CloudShell, which has the required AWS CLI preinstalled.
  • Make sure you are using AWS credentials with the required IAM permissions to create VPC Lattice constructs, update ECS services, and Route 53 records.
  • ECS must have the permissions to register and deregister targets into the VPC Lattice Target Group. To allow this, create an IAM Role that has the AmazonECSInfrastructureRolePolicyForVpcLattice AWS Managed Policy as permission policy attached to it. The role needs to allow the sts:AssumeRole Action for the principle ecs.amazonaws.com in its trusted identities, as described in the Amazon ECS infrastructure IAM role documentation.
  • The Security Group used by your ECS Services must allow Inbound traffic from the VPC Lattice prefix-list in the Region to the TCP Port used by your Service. The name of the prefix list can be found in the VPC Lattice Documentation in VPC Lattice using security groups – Managed Prefix Lists.
  • This walkthrough assumes that your existing internal ALB terminates TLS and also uses TLS to talk to the backends. We will migrate to TLS termination on the VPC Lattice services and TLS communication to the backends. If you are using cleartext HTTP, simply adjust the CLI examples to use port 80, HTTP protocol, and no certificates.

Step 1: VPC Lattice Target Group creation and mapping to ECS Services

We will look at the ECS Console and CLI workflows separately. At the time of writing, there are differences between the two. The ECS Console will create a new Target Group for you, but it cannot attach the Service to a pre-existing Target Group. The ECS CLI will only allow you to attach the ECS Service to a pre-existing Target Group.

Option 1: Using the ECS Console

  1. Navigate to the Amazon ECS Consoleand click on the link of the cluster running your services.
  2. Now scroll down and select the Service you want to attach to a new VPC Lattice Target Group as shown in Figure 4: Screenshot of ECS Services selection, then click on Update.
Figure 4: Screenshot of ECS Services selection

Figure 4: Screenshot of ECS Services selection

  1. Scroll down to the VPC Lattice section of the Service configuration, as show in Figure 5: VPC Lattice section in ECS Service configuration, and select the Turn on VPC Lattice radio button.
    Then enter the ARN of the Role you created for ECS to register targets into VPC Lattice Target Groups or select it in the drop-down list.
  2. Finaly, enter the Target Group Name that ECS will create for you and select the Port-Name of the containers in your Service that should be registered into the Target Group.

Figure 5: VPC Lattice section in ECS Service configuration

  1. Repeat the steps for every ECS Service you want to attach to a newly created VPC Lattice Target Group – In our example for the “backend-cart” and “backend-catalog” services.

Option 2: Using the ECS CLI

1. Create the VPC Lattice Target Groups

$ aws vpc-lattice create-target-group \
    --name backend-cart \
    --type IP \
    --config '{
        "port": 443,
        "protocol": "HTTPS",
        "protocolVersion": "HTTP1",
        "vpcIdentifier": "<Your VPC Id>"
        }'

$ aws vpc-lattice create-target-group \
    --name backend-catalog \
    --type IP \
    --config '{
        "port": 443,
        "protocol": "HTTPS",
        "protocolVersion": "HTTP1",
        "vpcIdentifier": "<Your VPC Id>"
        }'

Note that in your case you might have to supply additional detailed configuration for your health checks as the preceding CLI example only uses the default health check settings.

  1. Update the ECS Services to register the container IPs with the VPC Lattice Target Groups.
$ aws ecs update-service \
    --service backend-cart \
    --cluster ecs-cluster \
    --vpc-lattice-configurations '{
        "roleArn":"<ARN of Infrastructure Role>",
        "targetGroupArn":"<ARN of backend-cart Target Group>",
        "portName":"backend-cart"
        }'

$ aws ecs update-service \
    --service backend-catalog \
    --cluster ecs-cluster \
    --vpc-lattice-configurations '{
        "roleArn":"<ARN of Infrastructure Role>",
        "targetGroupArn":"<ARN of backend-catalog Target Group>",
        "portName":"backend-catalog"
        }'
  1. Verify that ECS has registered the Container IPs of your services to the Target Groups.
$ aws vpc-lattice list-targets \
    --target-group-identifier <Id of your Target Group>

{
    "items": [
        {
            "id": "10.0.4.222",
            "port": 443,
            "reasonCode": "TargetGroupNotInUse",
            "status": "UNUSED"
        }
    ]
}

The Status and reasonCode will show the Target Group as not in use until we attach it to a VPC Lattice
Service.

Step 2: Configure VPC Lattice

  1. Create the Service Network
$ aws vpc-lattice create-service-network \
    --name vpc-lattice-ecs
  1. Create a Security Group used for the Service Network VPC association
$ aws ec2 create-security-group \
    --group-name vpc-lattice \
    --description "Security Group for VPC Lattice SN association" \
    --vpc-id <Your VPC Id>

$ aws ec2 authorize-security-group-ingress \
    --group-id <Id of SG created in preceding command> \
    --protocol tcp \
    --port 443 \
    --source-group <Id of SG used by your ECS front-end Containers>
  1. Associate the Service Network to the VPC running the ECS front-end service
$ aws vpc-lattice create-service-network-vpc-association \
    --service-network-identifier <Id of Service Network> \
    --vpc-id <Your VPC Id> \
    --security-group-ids <Id of SG created in Step 2>
  1. Create the VPC Lattice Services for your back-end ECS Services
$ aws vpc-lattice create-service \
    --name backend-cart \
    --custom-domain-name cart.int.example.com \
    --certificate-arn <ARN of the certificate used for ALB listener>
$ aws vpc-lattice create-service \
    --name backend-catalog \
    --custom-domain-name catalog.int.example.com \
    --certificate-arn <ARN of the certificate used for ALB listener>
  1. Create the Listeners for your Services
$ aws vpc-lattice create-listener \
    --cli-input-json '{
        "name": "backend-cart",
        "protocol": "HTTPS",
        "port": 443,
        "serviceIdentifier": "<Id of the backend-cart Service>",
        "defaultAction": {
          "forward": {
            "targetGroups": [
              {
                "targetGroupIdentifier": "<Id of backend-cart TG>",
                "weight": 80
              }
            ]
          }
        }
    }'

$ aws vpc-lattice create-listener \
    --cli-input-json '{
        "name": "backend-catalog",
        "protocol": "HTTPS",
        "port": 443,
        "serviceIdentifier": "<Id of the backend-catalog Service>",
        "defaultAction": {
          "forward": {
            "targetGroups": [
              {
                "targetGroupIdentifier": "<Id of backend-catalog TG>",
                "weight": 80
              }
            ]
          }
        }
    }'
  1. Associate the Services to the Service Network
$ aws vpc-lattice create-service-network-service-association \
    --service-network-identifier <Id of the Service Network> \
    --service-identifier <Id of the backend-cart Service>

$ aws vpc-lattice create-service-network-service-association \
    --service-network-identifier <Id of the Service Network> \
    --service-identifier <Id of the backend-catalog Service>

Step 3: Validate routing through VPC Lattice

  1. Validate that the targets in the Target Groups are now showing up a HEALTHY
$ aws vpc-lattice list-targets \
       --target-group-identifier <Id of your Target Group>

   {
       "items": [
           {
               "id": "10.0.3.85",
               "port": 443,
               "status": "HEALTHY"
           }
       ]
    }
  1. (Optional) Validate that you are able to reach the back-end service through VPC Lattice through the auto assigned Service FQDN. Note: This will only be possible if your ECS Service was created or updated to allow the use of Amazon ECS Exec and has curl installed in the container image.
$ aws ecs execute-command \
    --cluster ecs-cluster \
    --task <Id of your frontend Task> \
    --interactive \
    --container frontend \
    --command "curl -H 'Host: cart.int.example.com' https://<auto assigned Service FQDN>"

Step 4: Update Route 53 records

  1. You will now shift traffic to VPC Lattice. For this, first navigate to the Route 53 Console
  2. Select Hosted Zones and click on the link to your private Hosted Zone holding the Alias records pointing to the internal ALB
  3. As shown in Figure 6: Route 53 record selection, select the record for the back-end service you want to shift to VPC Lattice and click on “Edit Record”

Figure 6: Route 53 record selection

  1. Edit the record
    1. Toggle the Alias switch to off
    2. Change the Record Type from A to CNAME
    3. Populate the Value with the auto generated FQDN of you VPC Lattice Service for the respective back-end
    4. Choose a short TTL initially, this allows you to fall back to the Alias record pointing to the internal ALB quickly if needed.

Figure 7: Route 53 record edit

  1. Repeat the step for every Route 53 record to finalize the shift from your internal ALB to VPC Lattice

Cleanup

  1. Update your Route 53 records with your desired TTL
  2. Check if your ALB still has any active connections
    1. Navigate to the EC2 Load Balancers page
    2. Select your internal ALB and select the Monitoring tab
    3. Check the Requests, Active Connection Count, and New Connection Count Those should all be and stay at zero since the shift to VPC Lattice happened. Do not proceed with the ALB deletion if the metrics are not at zero.
  3. Remove your internal ALB
    1. Navigate to the EC2 LoadBalancers page
    2. Select your internal ALB and select the Listeners and rules tab
    3. Select the Listener, then click on Manage rule and select Edit rules
    4. Select each rule pointing to your ECS Services which you migrated to VPC Lattice and click on Actions, select Delete rule, and confirm the deletion.
    5. Finaly, navigate back to the EC2 LoadBalancers page, select the internal ALB and click on Actions and select Delete load balancer, and confirm the deletion

Further considerations

  • If you were using a wildcard alias DNS record such as *.int.example.com pointing to ALB, you must create individual non-wildcard DNS records pointing to each individual auto-generated FQDNs of the VPC Lattice Services. At the time of this writing, VPC Lattice does not support custom wildcard domain names for services.
  • VPC Lattice supports both individual and wildcard certificates. Consult the Bring Your Own Certificate (BYOC) for VPC Lattice documentation page for details and caveats of using wildcard certificates with VPC Lattice.
  • If you have been using an internal Network Load Balancer (NLB) instead of ALB, you are able migrate to VPC Lattice if the protocol used to talk to your backends is supported by VPC Lattice. Consult the Listeners for your VPC Lattice service documentation page to learn what protocols are supported
  • Asses the impact of the migration If you were using Stickiness on the ALB Target groups. The Stickiness won’t be preserved during the migration to VPC Lattice.

Conclusion

In this post I discussed how to migrate your ECS Service to Service communication from an internal Application Load Balancer (ALB) to VPC Lattice with minimal downtime. To learn more about Amazon VPC Lattice, please refer to the documentation, and the AWS re:Post guide Get started with Amazon VPC Lattice – Resources and content. If you have questions about this post, then start a new thread on AWS re:Post, or contact AWS Support.

Further Reading

About the author

Yves Fauser

Yves Fauser

Yves is a Specialist Solution Architect at AWS. He is passionate about application networking and network automation, right where Dev meets Ops. Prior to joining AWS, Yves worked as an enterprise networking consultant, product manager, and solution architect, starting with physical networking and moving on to network virtualization and cloud networking.