Under the Hood: AWS CodeDeploy and Auto Scaling Integration

AWS CodeDeploy is a service that automates application deployments to your fleet of servers. Auto Scaling is a service that lets you dynamically scale your fleet based on load. Although these services are standalone, you can use them together for hands-free deployments! Whenever new Amazon EC2 instances are launched as part of an Auto Scaling group, CodeDeploy can automatically deploy your latest application revision to the new instances.

This blog post will cover how this integration works and conclude with a discussion of best practices. We assume you are familiar with CodeDeploy concepts and have completed the CodeDeploy walkthrough.

Configuring CodeDeploy with Auto Scaling

Configuring CodeDeploy with Auto Scaling is easy. Just go to the AWS CodeDeploy console and specify the Auto Scaling group name in your Deployment Group configuration.

In addition, you need to:

1. Install the CodeDeploy agent on the Auto Scaling instance. You can either bake the agent as part of the base AMI or use user data to install the agent during launch.
2. Make sure the service role used by CodeDeploy to interact with Auto Scaling has the correct permissions. You can use the AWSCodeDeployRole managed policy. For more information, see Create a Service Role for CodeDeploy.

For a step-by-step tutorial, see Using AWS CodeDeploy to Deploy an Application to an Auto Scaling Group.

Auto Scaling Lifecycle Hook

The communication between Auto Scaling and CodeDeploy during a scale in event is based on Auto Scaling lifecycle hooks. If the hooks are not set up correctly, the deployment will fail. We recommend that you do not try to manually set up or modify these hooks because CodeDeploy can do this for you. Auto Scaling lifecycle hooks tell Auto Scaling to send a notification when an instance is about to change to certain Auto Scaling lifecycle states. CodeDeploy listens only for notifications about instances that have launched and are about to be put in the InService state. This state occurs after the EC2 instance has finished booting, but before it is put behind any Elastic Load Balancing load balancers you have configured. Auto Scaling waits for a successful response from CodeDeploy before it continues working on the instance.

Hooks are part of the configuration of your Auto Scaling group. You can use the describe-lifecycle-hooks CLI command to see a list of hooks installed on your Auto Scaling group. When you create or modify a deployment group to contain an Auto Scaling group, CodeDeploy does the following:

1. Uses the CodeDeploy service role passed in for use with the deployment group to gain permissions to the Auto Scaling group.
2. Installs a lifecycle hook in the Auto Scaling group for instance launches that sends notifications to a queue owned by CodeDeploy.
3. Adds a record of the installed hook to the deployment group.

When you remove an Auto Scaling group from a deployment group or delete a deployment group, CodeDeploy does the following:

1. Uses the service role for the deployment group to gain access to the Auto Scaling group.
2. Gets the recorded hook from the deployment group and removes it from the Auto Scaling hook.
3. If the deployment group is being modified (not deleted), deletes the record of the hook from the deployment group.

If there are problems creating hooks, CodeDeploy will try to roll back the changes. If there are problems removing hooks, CodeDeploy will return the unsuccessful hook removals in the API response and continue.

Under the Hood

Here’s the sequence of events that occur during an Auto Scaling scale-in event:

1. Auto Scaling asks EC2 for a new instance.
2. EC2 spins up a new instance with the configuration provided by Auto Scaling.
3. Auto Scaling sees the new instance, puts it into Pending:Wait status, and sends the notification to Code Deploy.
4. CodeDeploy receives the instance launch notification from Auto Scaling.
5. CodeDeploy validates the configuration of the instance and the deployment group.
   1. If the notification looks correct, but the deployment group no longer contains the Auto Scaling group (or we can determine the deployment group was previously deleted) then CodeDeploy will not deploy anything and tell Auto Scaling to CONTINUE with the instance launch. Auto Scaling will respect any other constraints on instance launch; this step does not force Auto Scaling to continue if something else is wrong.
   2. If CodeDeploy can’t process the message (for example, if the stored service role doesn’t grant appropriate permissions), then CodeDeploy will let the hook time out. The default timeout for CodeDeploy is 10 minutes.
6. CodeDeploy creates a new deployment for the instance to deploy the target revision of the deployment group. (The target revision is the last successfully deployed revision to the deployment group. It is maintained by CodeDeploy.) You will need to deploy to your deployment group at least once for CodeDeploy to identify the target revision. You can use the get-deployment-group CLI command or the CodeDeploy console get the target revision for a deployment group.
   1. While the deployment is running, it sends heartbeats to Auto Scaling to let it know that the instance is still being worked on.
   2. If something goes wrong with the deployment, CodeDeploy will immediately tell Auto Scaling to ABANDON the instance launch. Auto Scaling terminates the instance and starts the process over again with a new instance.

Best Practices

Now that we know how the CodeDeploy and Auto Scaling integration works, let’s go over some best practices when using the two services together:

• Setting up or modifying Auto Scaling lifecycle hooks – We recommend that you do not try to set up or modify the Auto Scaling hooks manually because configuration errors could break the CodeDeploy integration.

• Beware of failed deployments – When a deployment to a new instance fails, CodeDeploy will mark the instance for termination. Auto Scaling will terminate the instance, spin up a new instance, and notify CodeDeploy to start a deployment. This is great when you have transient errors. However, the downside is that if you have an issue with your target revision (for example, if there is an error in your deployment script), this cycle of launching and terminating instances can go into a loop. We recommend that you closely monitor deployments and set up Auto Scaling notifications to keep track of EC2 instances launched and terminated by Auto Scaling.

• Troubleshooting Auto Scaling deployments – Troubleshooting deployments involving Auto Scaling groups can be challenging. If you have a failed deployment, we recommend that you disassociate the Auto Scaling group from the deployment group to prevent Auto Scaling from continuously launching and terminating EC2 instances. Next, add a tagged EC2 instance launched with the same base AMI to your deployment group, deploy the target revision to that EC2 instance, and use that to troubleshoot your scripts. When you are confident, associate the deployment group with the Auto Scaling group, deploy the golden revision to your Auto Scaling group, scale up a new EC2 instance (by adjusting Min, Max, and Desired values), and verify that the deployment is successful.

• Ordering execution of launch scripts – The CodeDeploy agent looks for and executes deployments as soon as it starts. There is no ordering between the deployment execution and   launch scripts such as user data, cfn-init, etc. We recommend you install the host agent as part of (and maybe as the last step in) the launch scripts so that you can be sure the deployment won’t be executed until the instance has installed dependencies that are not part of your CodeDeploy deployment. If you prefer baking the agent into the base AMI, we recommend that you keep the agent service in a stopped state and use the launch scripts to start the agent service.

• Associating multiple deployment groups with the same Auto Scaling group – In general, you should avoid associating multiple deployment groups with the same Auto Scaling group. When Auto Scaling scales up an instance with multiple hooks associated with multiple deployment groups, it sends notifications for all of the hooks at the same time. As a result, multiple CodeDeploy deployments are created. There are several drawbacks to this. These deployments are executed in parallel, so you won’t be able to depend on any ordering between them. If any of the deployments fail, Auto Scaling will immediately terminate the instance. The other deployments that were running will start to fail when the instance shuts down, but they may take an hour to time out.  The host agent processes only one deployment command at a time, so you have two more limitations to consider. First, it’s possible for one of the deployments to be starved for time and fail. This might happen, for example, if the steps in your deployment take more than five minutes to complete. Second, there is no preemption between deployments, so there is no way to enforce step ordering between one deployment and another. We therefore recommend that you minimize the number of deployment groups associated with an Auto Scaling group and consolidate the deployments into a single deployment.

We hope this deep dive into the Auto Scaling integration with CodeDeploy gives you the insight needed to use it effectively.  Are there other features or scenarios with CodeDeploy that you’d be interested in understanding the inner details better?  Let us know in the comments.