Automating Amazon EBS snapshot and AMI management using Amazon DLM

Many AWS customers frequently look for ways to lower operational costs and complexity of managing their backup operations. With Amazon EBS, you can create EBS snapshots and EBS-backed AMIs that are a convenient way to back up your block level data, regardless of where it resides. EBS snapshots are a block-level, point-in-time, incremental copies of your Amazon EBS volumes for backup, or disaster recovery (DR) purposes. Your organization’s IT policy may stipulate that snapshots be taken on a specified schedule (hourly, daily, weekly, etc.) to meet business continuity and data protection requirements. In addition, you may have compliance guidelines to copy snapshots across AWS Regions, apply prescribed retention policies, and perform regular recovery actions as part of disaster readiness.

Previously, you may have invested in tools to automate the creation of Amazon EBS snapshots by writing custom scripts, or used Amazon CloudWatch rules for EBS volume resources. This would involve manually managing the retention and deletion of EBS snapshots according to your recovery point objectives (RPO).

In this blog post, we examine how you can use Amazon Data Lifecycle Manager (Amazon DLM) lifecycle policies to automate the creation, retention, and deletion of Amazon EBS snapshots. With Amazon DLM, the need for complicated and custom scripts to manage EBS snapshots is eliminated. Amazon DLM enables you to create, manage, and delete EBS snapshots in a simple, automated way based on resource tags for EBS volumes or Amazon EC2 instances. This reduces the operational complexity of managing EBS snapshots, thereby saving time and money. Also, let’s not forget the best part: Amazon DLM is free to use and is available in all AWS Regions.

It has been almost a year since we published the original version of this blog. Since then, we have added functionality that allows you to add up to four schedules per policy, to enable automatic cross-account sharing and copying of snapshots, and to automate the lifecycle of EBS-backed AMIs.

Getting started with Amazon DLM

To get started, launch the AWS Management Console, then select Lifecycle Manager under the Elastic Block Store navigation area of the Amazon EC2 dashboard. As shown in the following screenshot, you can create a new lifecycle policy under Create new lifecycle Policy. Note that this policy applies only to resources within the selected account for the selected AWS Region. For resources in other accounts or other AWS Regions, you must create a Region-specific Amazon DLM policy for each account.

Defining backup policies and resource tagging

Before creating your Amazon DLM snapshot policy, you must make sure that the resource tags are already assigned to EBS volumes and EC2 instances. You can specify tags to existing resources or apply at the time of creating a new resource. Many customers first define recovery point objectives (RPO) and recovery time objectives (RTO) for different tier applications based on business requirements. Each tier may have unique requirements for snapshot creation, retention, and copy across Regions. Here’s one such example policy definition document template for your organization:

In the Tier 1 example, Amazon DLM will create a snapshot every hour, and retain each snapshot for 24 hours before deleting the snapshot. Each snapshot will have Amazon EBS fast snapshot restore enabled, and will also be copied to another Region, where it is kept for 2 days before being deleted.

Creating an EBS snapshot policy

Now that you have an Amazon DLM policy document for your organization, you can create a snapshot lifecycle policy and schedule. After selecting the policy type as EBS snapshot policy, you can then select the resource of (Amazon EBS) Volume or (Amazon EC2) Instance as shown in the following screenshot, and apply the resource tags to this policy. This policy is applied to all EBS volumes with any of the assigned tags. You may use the same tag Key and Value for two different Amazon DLM policies.

In this example, we selected Volume as the resource type. However, by selecting the other option of Instance, you can create a crash-consistent set of snapshots for all the EBS volumes attached to an instance targeted with the Key and Value tag assignments. A crash-consistent snapshot ensures that your data is coordinated and consistent across all the volumes that are attached to an instance when you take a backup.

Next, set the Policy status to Enabled. The initial snapshot creation is initiated within an hour from the schedule you defined.

For the next step, define a snapshot schedule for the policy. You can either use cron expressions, or specify hourly, daily, weekly, monthly, and annual schedules.

Continuing with the preceding example, we create a Policy Schedule and set the Frequency to Daily. We further specify that the frequency should be hourly and start at a specified time of the day. Next, we specify Retention type based on the Age of the snapshot. Amazon DLM also supports retention based on the number of snapshots you like to keep by selecting count-based Retention type.

Cross-Region copy through Amazon DLM

You can also automate the copying of a snapshot to another AWS Region after it has been created. Copies can be scheduled for up to three AWS Regions from a single policy, and retention periods are set for each AWS Region separately. The ability to copy snapshots using Amazon DLM helps simplify backup and DR workflows by providing an automated way to manage the creation and retention of snapshot copies.

Amazon DLM ensures that cross-Region copies are incremental to minimize data transfer and optimize snapshot usage. There are certain considerations when copying encrypted snapshots of your EBS volumes. For more details, see permissions for encrypted snapshots.

For this example, we select the Enable cross-Region copy for this schedule check box to enable Cross-Region copy. Then, we select a Target Region, and apply snapshot retention as 2 days after Creation. Next, we select Encryption and choose the KMS key to use to encrypt the copies in the destination Region. 

Creating an Amazon EBS-backed AMI policy for Amazon EC2 instances

Amazon DLM allows you to create policies that automate the creation, copy and deregistration of EBS-backed AMIs. Furthermore, once AMIs are deregistered, DLM will automatically delete any snapshots backing the AMI.

To get started, choose EBS-backed AMI policy under Create new lifecycle policy and create the policy as you would for an Amazon EBS snapshot policy.

Automating sharing and cross-account copy through Amazon DLM

You can also automatically share snapshots created by a policy with other AWS accounts. For more information on automating the sharing and cross-account copy of snapshots with Amazon DLM, visit this blog post.

Monitoring Amazon DLM snapshot actions

Amazon DLM emits EBS snapshot lifecycle events to the AWS CloudTrail console on the Event history page. You can look up events related to creation or deletion of snapshots under the User name filter DataLifecycleManager, as shown in the screenshot.

Cleaning up

If you followed along and created test Amazon EBS volumes and Amazon DLM policies to create EBS snapshots, and also copied EBS snapshots across AWS Regions, be sure to clean up all unwanted resources to avoid unnecessary charges.

Conclusion

In summary, automating the lifecycle of Amazon EBS snapshots and EBS-backed AMIs using Amazon DLM helps you manage your EBS resources efficiently, thereby reducing your costs and management complexity. You can protect valuable data by enforcing a regular backup schedule, in addition to reduce storage costs by deleting outdated backups. Combined with the monitoring features of Amazon CloudWatch Events and AWS CloudTrail, Amazon DLM provides a complete backup solution for your EBS resources at no additional cost. For more information about Amazon Data Lifecycle Manager, visit the service documentation.

Thank you for reading this blog post! If you have any comments or questions, please don’t hesitate to leave them in the comments section.