Automating Amazon EBS snapshot management using Data Lifecycle Manager

UPDATE: An updated version of this post was published on 7/14/2021. Please refer to that post for the most up-to-date content.

Many AWS customers frequently look for ways to lower the operational costs and complexity of managing their backup operations. With Amazon EBS, you can create snapshots that are a convenient way to back up your block level data, regardless of where it resides. EBS snapshots are a block-level, point-in-time, incremental copies of your Amazon EBS volumes for backup, or disaster recovery (DR) purposes. Your organization’s IT policy may stipulate that EBS snapshots be taken on a specified schedule (hourly, daily, weekly, etc.) to meet business continuity and data protection requirements. In addition, you may have compliance guidelines to copy EBS snapshots across AWS Regions, apply prescribed retention policies, and perform regular recovery actions as part of disaster readiness.

Previously, you may have invested in tools to automate the creation of Amazon EBS snapshots by writing custom scripts, or used Amazon CloudWatch rules for EBS volume resources. This would involve manually managing the retention and deletion of EBS snapshots according to your recovery point objectives (RPO).

In this blog post, we examine how you can use Amazon Data Life Cycle Manager (Amazon DLM) lifecycle policies to automate the creation, retention, and deletion of Amazon EBS snapshots. With Amazon DLM, the need for these complicated and custom scripts to manage EBS snapshots is eliminated. Amazon DLM enables you to create, manage, and delete EBS snapshots in a simple, automated way based on resource tags for EBS volumes or EC2 instances. This reduces the operational complexity of managing EBS snapshots, thereby saving time and money. Also, let’s not forget the best part: Amazon DLM is free to use and is available in all AWS Regions.

Getting started with Amazon DLM

To get started, you launch the AWS Management Console, and then select Lifecycle Manager under the Elastic Block Store navigation area of the Amazon EC2 dashboard. As shown in the following screenshot, you can Create Snapshot Lifecycle Policy. Note that this policy applies only to resources within the selected account for the selected AWS Region. For resources in other accounts or other AWS Regions, you must create a Region-specific Amazon DLM policy for each account.

Defining backup policies and resource tagging

Before creating your Amazon DLM snapshot policy, you must make sure that the resource tags are already assigned to EBS volumes and EC2 instances. You can specify tags to existing resources or apply at the time of creating a new resource. Many customers first define recovery point objectives (RPO) and recovery time objectives (RTO) for different tier applications based on business requirements. Each tier may have unique requirements for snapshot creation, retention, and copy across Regions. Here’s one such example policy definition document template for your organization:

Creating a snapshot lifecycle policy

Now that you have an Amazon DLM policy document for your organization, let us create a snapshot lifecycle policy and schedule. You select the resource of (Amazon EBS) Volume or (Amazon EC2) Instance as shown in the following screenshot, and apply the resource tags to this policy. This policy is applied to all EBS volumes with any of the assigned tags. You may use the same tag Key and Value for two different Amazon DLM policies.

In this example, we selected Volume as the resource type. However, by selecting the other option of Instance, you can create a crash-consistent set of snapshots for all the EBS volumes attached to an instance targeted with the Key and Value tag assignments. A crash-consistent snapshot ensures that your data is coordinated and consistent across all the volumes that are attached to an instance when you take a backup.

For the next step, define a snapshot schedule for the preceding policy. Recently we launched support for cron expressions as the input for your Amazon DLM schedules. Cron expressions provide you the flexibility to specify custom schedules in your policies. With this launch, we have also expanded the backup intervals on Amazon DLM to support hourly, daily, weekly, monthly, and annual schedules.

Continuing with the preceding example, we create a Policy Schedule and set the Frequency to Daily. We further specify that the frequency should be hourly and start at a specified time of the day. Next, we specify Retention type based on the Age of the snapshot. Amazon DLM also supports retention based on the number of snapshots you like to keep by selecting count-based Retention type.

Cross-Region copy through Amazon DLM

Last year we launched the capability to copy snapshots across AWS Regions using Amazon DLM. Copies can be scheduled for up to three AWS Regions from a single policy, and retention periods are set for each AWS Region separately. The ability to copy snapshots using Amazon DLM helps simplify backup and DR workflows by providing an automated way to manage the creation and retention of snapshot copies.

Amazon DLM ensures that cross-Region copies are incremental to minimize data transfer and optimize snapshot usage. There are certain considerations when copying encrypted snapshots of your EBS volumes. For more details, see permissions for encrypted snapshots.

For this example, we click the Enable check box to Enable cross region copy. Then, we select a Target region, and apply Snapshot retention as 2 days After Creation. Next, we select Encryption and supply the AWS KMS Master Key from the drop-down.

Enable fast snapshot restore (FSR) through Amazon DLM

Last year, we launched the capability to enable Amazon EBS fast snapshot restore (FSR) on EBS snapshots and integrated it with Amazon DLM. EBS volumes created from FSR-enabled EBS snapshots are fully initialized upon creation and immediately deliver all provisioned performance.

In this example, we select the box to Enable fast snapshot restore and set a Duration of 1 (day) retention for the FSR state in select Availability Zones. This ensures that FSR is automatically disabled on EBS snapshots after the specified time period elapses for better cost management.

Finally, enable Policy status after creation by selecting Enable policy. The initial snapshot creation is executed within an hour from the schedule you defined.

Monitoring Amazon DLM snapshot actions

Amazon DLM emits Amazon EBS snapshot lifecycle events to the AWS CloudTrail console on the Event history page. You can look up events related to creation or deletion of EBS snapshots under the User name filter DataLifecycleManager, as shown in the screenshot.

Cleaning up

As a part of this example, if you have created any test EBS volumes and Amazon DLM policies to create EBS snapshots, applied FSR, and copied EBS snapshots across AWS Region, be sure to clean up all unwanted resources to avoid unnecessary charges.

Conclusion

In summary, automating snapshot lifecycle using Amazon DLM helps you manage your Amazon EBS snapshots efficiently, thereby reducing your costs and management complexity. You can protect valuable data by enforcing a regular backup schedule, in addition to reduce storage costs by deleting outdated backups. Combined with the monitoring features of Amazon CloudWatch Events and AWS CloudTrail, Amazon DLM provides a complete backup solution for EBS volumes at no additional cost. Use this step-by-step guide to create snapshot lifecycle policies and start automating the creation, retention, and deletion of EBS snapshots.

Thank you for reading this blog post! For more information about Amazon EBS, visit the Amazon EBS product page. If you have any comments or questions, please don’t hesitate to leave them in the comments section.