Updating AWS Global Accelerator EC2 endpoints automatically based on Auto Scaling group events

AWS Global Accelerator is a network layer service that directs traffic to optimal endpoints over the AWS global network, this improves the availability and performance of your internet applications that are used by a global audience. It provides static IP addresses that act as a fixed entry point to your application endpoints in a single or multiple AWS Regions, such as your Application Load Balancers, Network Load Balancers, Elastic IP addresses or Amazon EC2 instances. To front an EC2 instance with Global Accelerator, you create an accelerator and add the EC2 instance as an endpoint using the EC2 instance ID.

Some customers use the AWS Auto Scaling service to automatically adjust capacity to maintain steady, predictable performance at the lowest possible cost. A Load Balancer can be attached to the Auto Scaling Group and used as endpoint for the endpoint group of the accelerator. However, there are use cases that require adding the EC2 instances as endpoints directly to the endpoint group. An example is when you want to send UDP traffic with client IP preservation to a handful of instances, with a guarantee that the same backend instances will handle requests from the same clients (client affinity). This is not possible with Application Load Balancers because they do not support UDP traffic, and Network Load Balancers do not support sticky sessions or client IP preservation with AWS Global Accelerator.

In this post, I show you how to use AWS Lambda to automatically add EC2 endpoints to a Global Accelerator endpoint group, or remove EC2 endpoints from an endpoint group based on Auto Scaling group events. We recommend that you remove an EC2 instance from Global Accelerator endpoint groups before you terminate the instance. If you terminate an EC2 instance before you remove it from an endpoint group in Global Accelerator, and then you create another instance in the same VPC with the same private IP address, and health checks pass, Global Accelerator will route traffic to the new endpoint. We use Auto Scaling lifecycle hooks to remove instances selected for termination from the endpoint group before they are terminated.

Solution overview

The following diagram describes the solution at a high level:

An AWS Global Accelerator endpoint has EC2 endpoints that are launched by an Auto Scaling group.
The Auto Scaling group terminates or launches EC2 instances.
An Amazon CloudWatch Events rule captures these events and triggers a Lambda function.
The Lambda function updates the Accelerator endpoint using the UpdateEndpointGroup API.

Prerequisites and deployments options

Make sure you have the following completed:

Your Auto Scaling group is created and configured. Make note of the name.
Your accelerator and the endpoint group are created. Make note of the endpoint group Amazon Resource Name (ARN).

To implement this solution, you create the following:

An Identity and Access Management (IAM) role.
A Lambda function.
A CloudWatch Events rule.

We recommend using AWS CloudFormation to automatically create and configure the different resources (IAM role, Lambda function, and CloudWatch rule). The CloudFormation template with the inline Lambda function code is available in the first part of this post (automated deployment). In the second part (manual deployment), I provide how to implement the solution step by step using the AWS Management Console and the AWS Command Line Interface (CLI), in case you are more comfortable with the CLI.

Automated deployment using AWS CloudFormation

AWS CloudFormation gives you an easy way to codify the creation and management of related AWS resources. The optional Mappings and Parameters sections of CloudFormation templates help you organize and parameterize your templates so you can quickly customize your stack. The step-by-step instructions in this section show you how you can automate the creation and configuration of resources needed for the solution (IAM role, Lambda function, and CloudWatch rule). After you choose Create stack, the resources are created in three to four minutes.

Choose Launch stack. The template is loaded from Amazon S3 automatically. The template is launched in the US East (N. Virginia) AWS Region by default. To launch the CloudFormation stack in a different AWS Region, use the region selector in the console navigation bar after you choose Launch stack. Choose Next.
On the Specify stack details page, change the stack name, if needed (default is GlobalAcceleratorAndAutoScaling). Provide the following values to the CloudFormation template and choose Next. These (user input) values will be passed as parameters to the template.
1. Endpoint group ARN (required)
2. Endpoint weight (optional)
3. Auto Scaling group name (required)
On the Configure stack options page, add tags if needed (not required), then choose Next.
On the Review stack page, select I acknowledge that AWS CloudFormation might create IAM resources, then choose Create stack.

You can also download the template and use it as a starting point for your own implementation.

Corresponding CLI command

Download the template and run the following command, making sure that you enter the correct Global Accelerator endpoint group ARN and Auto Scaling group name.

$aws cloudformation create-stack \
--stack-name GlobalAcceleratorAndAutoScaling \
--template-body file://asg_aga.yaml \
--parameters ParameterKey=EndpointGroupARN,ParameterValue=<YOUR-ENDPOINT-GROUP-ARN> \
ParameterKey=AutoscalingGroupName,ParameterValue=<YOUR-AUTOSCALING-GROUP-NAME>
--region <AWS-REGION>

Manual deployment using AWS Web Management console or AWS CLI

Step 1 – Create and configure the Lambda function’s IAM role.

Lambda functions need an IAM role to give them their execution permissions. To create the IAM role, complete the following steps. For more information, see Creating an IAM Role.

In the IAM console, choose Policies, Create Policy.
On the JSON tab, enter the following IAM policy (make sure you replace <YOUR-ENDPOINT-GROUP-ARN>):

{
    "Version": "2012-10-17",
    "Statement": [{
            "Effect": "Allow",
            "Action": [
                "autoscaling:CompleteLifecycleAction"
            ],
            "Resource": ""
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:::"
        },
        {
            "Effect": "Allow",
            "Action": [
                "globalaccelerator:UpdateEndpointGroup",
                "globalaccelerator:DescribeEndpointGroup"
            ],
            "Resource": "<YOUR-ENDPOINT-GROUP-ARN>"
        }
    ]
}

Choose Review policy.
Enter a name (ASG_AGA-Lambda-Policy) for this policy, and choose Create policy. Note the name of this policy for later steps.
In the left navigation pane, choose Roles, Create role.
On the Select role type page, choose AWS Service and then Lambda to allow Lambda functions to call AWS services on your behalf.
Choose Next: Permissions.
Filter policies by the policy name that you just created (ASG_AGA-Lambda-Policy), and check the box.
Choose Next: Tags, and give it an appropriate tag.
Choose Next: Review. Give this IAM role an appropriate name (ASG_AGA-Lambda-Role), and note it for future use.
Choose Create role.

Corresponding CLI commands

Create a text file called Lambda-Role-Inline-Policy.json with the preceding policy as content.

Create a text file called Lambda-Role-Trust-Policy.json with the following content:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}

Create a policy with this trust policy, and then attach the inline policy to it:

$ aws iam create-role \
    --role-name ASG_AGA-Lambda-Role \
    --assume-role-policy-document file://Lambda-Role-Trust-Policy.json
    
$ aws iam put-role-policy \
    --role-name ASG_AGA-Lambda-Role \
    --policy-name ASG_AGA_Lambda_CW \
    --policy-document file://Lambda-Role-Inline-Policy.json

Step 2 – Put the lifecycle hook for instance terminating.

It is recommended to remove EC2 endpoints from Accelerator endpoint groups before they are terminated. When using AWS Auto Scaling you don’t have control over instance terminations. Use the endpoint group before it is terminated. To create the lifecycle hook, take the following steps:

In the Auto Scaling console, select your Auto Scaling Group.
Select the Lifecycle Hooks tab, and then choose Create Lifecycle Hook.
Give a name to the lifecycle hook.
For Lifecycle Transition, select Instance Terminate.
For Heartbeat Timeout, type 60.
For Default Result, keep ABANDON.
Create the lifecycle hook.

Corresponding CLI command

$ aws autoscaling put-lifecycle-hook \
    --lifecycle-hook-name ASG-AGA-Terminating \
    --auto-scaling-group-name ASG-Group-Name \
    --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
    --heartbeat-timeout 60

Step 3 – Create the Lambda function.

To create a Lambda function, complete the following steps. For more information, see Create a Lambda Function with the Console.

In the Lambda console, choose Author from scratch.
For Function Name, enter the name of your function (ASG_AGA-Lambda-Function for example).
For Runtime, choose Python 3.8.
Under Permissions, for Execution role, select Use an existing role, then select the IAM role created in the previous step (ASG_AGA-Lambda-Role).
Choose Create Function, remove the default function, and copy the following code into the Function Code window:

'''
# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file.
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
#
# Description: This Lambda function updates an AWS Global Accelerator Endpoint Group based on Auto Scaling group Events.
'''
import boto3
import hashlib
import json
import logging
import os
import time

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

aga_client = boto3.client('globalaccelerator', region_name='us-west-2')

EC2_LAUNCHING = 'EC2 Instance Launch Successful'
EC2_TERMINATING = 'EC2 Instance-terminate Lifecycle Action'
ENDPOINT_GROUP_ARN = os.environ['EndpointGroupARN']

if (os.environ.get('EndpointWeight') != None) and os.environ['EndpointWeight'].isdigit() and int(os.environ['EndpointWeight']) < 256:
    ENDPOINT_WEIGHT = int(os.environ['EndpointWeight'])
else:
    ENDPOINT_WEIGHT = 128

def check_response(response_json):
    if response_json.get('ResponseMetadata', {}).get('HTTPStatusCode') == 200:
        return True
    else:
        return False

def list_endpoints():
    response = aga_client.describe_endpoint_group(
        EndpointGroupArn = ENDPOINT_GROUP_ARN
    )
    return response

def updated_endpoints_list(detail_type, instance_id):
    endpoints = []
    response = list_endpoints()

    if detail_type == EC2_LAUNCHING:
        for EndpointID in response['EndpointGroup']['EndpointDescriptions']:
            result = {'EndpointId': EndpointID['EndpointId'],'Weight': EndpointID['Weight']}
            endpoints.append(result)
        endpoints.append({'EndpointId': instance_id,'Weight': ENDPOINT_WEIGHT}) # Add the endpoint

    elif detail_type == EC2_TERMINATING:
        for EndpointID in response['EndpointGroup']['EndpointDescriptions']:
            if EndpointID['EndpointId'] != instance_id: # Remove the endpoint
                result = {'EndpointId': EndpointID['EndpointId'],'Weight': EndpointID['Weight']}
                endpoints.append(result)
    return endpoints

def update_endpoint_group(detail_type, instance_id):
    try:
        response = aga_client.update_endpoint_group(
            EndpointGroupArn = ENDPOINT_GROUP_ARN,
            EndpointConfigurations = updated_endpoints_list(detail_type, instance_id)
            )
        if check_response(response):
            logger.info("The endpoint group has been updated: %s", response)
            return response['EndpointGroup']['EndpointDescriptions']
        else:
            logger.error("Could not update the endpoint group: %s", response)
            return None
    except Exception as e:
        logger.error("Could not update the endpoint group: %s", str(e))
        return None

def lambda_handler(event, context):
    try:
        logger.info(json.dumps(event))
        message = event['detail']
        detail_type = event['detail-type']
        if 'AutoScalingGroupName' in message:
            instance_id = message['EC2InstanceId']
            response = update_endpoint_group(detail_type, instance_id)
            if response != None:
                logging.info("Lambda executed correctly")
            if detail_type == EC2_TERMINATING: # Abandon the lifecycle hook action
                asg_client = boto3.client('autoscaling')
                abandon_lifecycle = asg_client.complete_lifecycle_action(
                    LifecycleHookName = message['LifecycleHookName'],
                    AutoScalingGroupName = message['AutoScalingGroupName'],
                    LifecycleActionResult = 'ABANDON',
                    InstanceId = instance_id
                    )
                if check_response(abandon_lifecycle):
                    logger.info("Lifecycle hook abandoned correctly: %s", response)
                else:
                    logger.error("Lifecycle hook could not be abandoned: %s", response)
        else:
            logging.error("No valid JSON message: %s", parsed_message)
    except Exception as e:
        logging.error("Error: %s", str(e))

In the Environment variables section, enter the following key-value pair(s):
1. Key = EndpointGroupARN | Value = the ARN of the Accelerator Endpoint Group
2. Optional – To override the default Endpoint weight (128), add the following key-value pair: Key = EndpointWeight | Value = the endpoint weight you would like for new EC2 instances
In Basic settings, increase the Timeout from 3 to 30 seconds.
In Concurrency section, select Reserve concurrency and enter 1.
Save.

Corresponding CLI commands

Create a text file called asg_aga_function.py with the preceding Python code as content. Zip it and use the following CLI command to create a Lambda function with the name ASG_AGA-Function. The function uses the IAM role created in step 1 (ASG_AGA-Lambda-Role). Set the timeout to 30 seconds and the maximum number of simultaneous executions to 1.

$ zip asg_aga_function.zip asg_aga_function.py

$ aws lambda create-function \
    --function-name ASG_AGA-Function \
    --runtime Python 3.8 \
    --zip-file fileb://asg_aga_function.zip \
    --role arn:aws:iam::123456789012:role/ASG_AGA-Lambda-Role \
    --handler asg_aga_function.handler \
    --timeout 30

$ aws put-function-concurrency \
    --function-name ASG_AGA-Function
    --reserved-concurrent-executions 1

Step 4 – Create and configure the CloudWatch Events rule.

The rule catches and triggers the Lambda function (set as a target) every time the Auto Scaling group launches an EC2 instance (EC2 Instance Launch Successful event) or terminates an EC2 instance. For this, you must make sure the EC2 endpoint is removed from the accelerator endpoint before it is terminated. Use the lifecycle hook for this (EC2 Instance-terminate Lifecycle Action event).

To create a CloudWatch Events rule, complete the following steps:

In the CloudWatch console, choose Rules, Create rule.
Under Event Source, select Event Pattern.
Choose Edit next to Event Pattern Preview.
Copy the following JSON into the preview pane (make sure you replace ASG-Group-Name with the name of your Auto Scaling group):

{
    "source": ["aws.autoscaling"],
    "detail-type": ["EC2 Instance Launch Successful", "EC2 Instance-terminate Lifecycle Action"],
    "detail": {
        "AutoScalingGroupName": ["ASG-Group-Name"]
    }
}

For Targets, select Lambda function, and select the Lambda function created in step 3 (ASG_AGA-Lambda-Function in our sample).
Choose Configure details.
Enter a name (ASG-AGA-Rule) and description for the rule.
For State, select Enabled.
Choose Create rule.

Corresponding CLI commands

Create a text file called eventPattern.json with the preceding JSON content (make sure you replace ASG-Group-Name with the name of your Auto Scaling group). Create the rule, and then add the Lambda function as target for it as follows (update the region and the AWS account number in the function ARN):

$ aws events put-rule \
    --name ASG-AGA-Rule \
    --event-pattern file://eventPattern.json

$ aws events put-targets
    --rule ASG-AGA-Rule \
    --targets "Id"="1","Arn"="arn:aws:lambda:us-west-2:123456789012:function:ASG_AGA-Function"

Test the environment

In the Auto Scaling console, select your Auto Scaling Group.
In Details tab, select Edit.
Change the desired and minimum capacity to 0. If any of these instances were endpoints for the accelerator endpoint group, you will notice that before they are terminated. They will be removed from the accelerator endpoint group.

Corresponding CLI commands

$ aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name <ASG-Group-Name> \
    --min-size 0 --desired-capacity 0

$ aws globalaccelerator describe-endpoint-group \
    --endpoint-group-arn <YourEndpointGroupARN> \
    --region us-west-2

Change back the Auto Scaling group desired and minimum capacity to the ones expected, verify that the new EC2 instances are automatically added to the accelerator endpoint group as soon as they are successfully launched.

If it does not work as expected, review the CloudWatch Logs to see the Lambda output. In the CloudWatch console, choose Logs and /aws/lambda/ASG_AGA-Function to see the execution output.

Conclusion

In this post, I demonstrated how you can use CloudWatch Event rules and AWS Lambda to automatically update AWS Global Accelerator EC2 endpoints based on AWS Auto Scaling Groups events.

It would be great to hear about how you are using this function. Do not hesitate to leave me your questions and comments.

About the Author

Jibril Touzi is a Technical Account Manager at AWS. Helping partners and customers to innovate using AWS Networking and Edge services is what keeps him motivated. Jibril is a passionate photographer; when he is not working, he enjoys spending time with family in outdoor activities.

		Blog: Using AWS Client VPN to securely access AWS and on-premises resources
		Learn about AWS VPN services
		Watch re:Invent 2019: Connectivity to AWS and hybrid AWS network architectures

Networking & Content Delivery

Updating AWS Global Accelerator EC2 endpoints automatically based on Auto Scaling group events

Solution overview

Prerequisites and deployments options

Automated deployment using AWS CloudFormation

Corresponding CLI command

Manual deployment using AWS Web Management console or AWS CLI

Step 1 – Create and configure the Lambda function’s IAM role.

Corresponding CLI commands

Step 2 – Put the lifecycle hook for instance terminating.

Corresponding CLI command

Step 3 – Create the Lambda function.

Corresponding CLI commands

Step 4 – Create and configure the CloudWatch Events rule.

Corresponding CLI commands

Test the environment

Corresponding CLI commands

Conclusion

About the Author

Resources

Follow