Simple and comprehensive data protection with Amazon Data Lifecycle Manager

Enterprises often use distinct accounts to group workloads and associated resources used across multiple teams and projects. This helps organizations align ownership, decision making, and costs so that they can be easily managed across internal teams. However, each team in an account may have different requirements and processes when it comes to backing up their critical and non-critical resources. This mixture of requirements and processes can lead to missed backups that result in lost data and cause significant downtime, jeopardizing the Recovery Point Objective. Meanwhile, account owners and administrators do not know the details of each team and project’s processes and thus find it challenging to enforce account-level backup requirements.

Customers today have the choice of using a wide range of products, features, and custom scripts to automatically back up their Amazon Elastic Block Store (Amazon EBS) volumes into EBS Snapshots and Amazon Elastic Compute Cloud (Amazon EC2) into EBS-backed Amazon Machine Images (EBS-backed AMIs). These tools operate independently of one another, and duplicate backups can be created if different teams employ different tools to back up the same resource, incurring additional costs.

Account owners and storage administrators can now use Amazon Data Lifecycle Manager default policies to protect all critical workloads while minimizing the number of resources that are created. A default policy only creates new backups for instances or volumes when they don’t have recent backups that meet the creation frequency set in the default policy. Otherwise, if backups are being successfully created by any other mechanism, then the default policy does not create new AMIs or snapshots for the targeted resource(s).

In this post, we walk through how customers can setup Amazon Data Lifecycle Manager default policies to make sure critical workloads are backed up while minimizing duplicate resource creation and cost. We outline the setup of default policies for EC2 instances and EBS volumes and validate the resources your default policies create in the Amazon EC2 console. Default policies are also a great option for customers who want a simple way to enable automated backups of all the instances and volumes in their account. Using default policies gives account owners and administrators peace-of-mind that all critical workloads in their account are backed up regardless of the different processes used by teams and projects that operate in the account.

Scenarios

The following are two common scenarios where backups are required:

Scenario 1: A single user wants a simple way to backup all resources

Alice is the owner of an account who wants a simple way to enable automated backups of all the instances and volumes in her account so that they are guaranteed to have a backup every day. They want to be able to enable the feature through the EC2 console and know it will ‘just work.’

Scenario 2: The storage administrator wants to make sure all critical workloads are backed up

Bob is a storage administrator for his organization and is responsible for making sure all critical workloads in key accounts are regularly backed up. The accounts are used by multiple teams and users, each with their own requirements. If any of the accounts’ users are not able to find backups to restore data from, then Bob is their first point of contact. The following diagram is an example of all the teams and workloads that operate in a single account.

• Team A runs critical applications on volumes 1-4 (Amazon EBS io2 Block Express volumes) and needs hourly backups.
• Team B runs critical applications on volumes 5-7 (Amazon EBS io2 Block Express volumes) and needs hourly backups of only data volumes.
• Team B stores temporary data on volumes 8 (Amazon EBS gp3 volume) and does not need any backups.
• Team C runs critical applications on volumes 9-10 (Amazon EBS io1 volume, io2 Block Express volume) and needs hourly backups of only data volumes.
• Individual users create volumes 11-14 (Amazon EBS io2 Block Express volume) and must adhere to the organization’s hourly backup requirement, except if they are used for testing purposes.

Each team also uses their own mechanism to back up their data:

• Team A uses a third-party tool targeting resources to back up based on volume tags.
• Team B uses Amazon Data Lifecycle Manager custom policies targeting resources to back up based on volume tags
• Team C users custom scripts targeting resources to back up based on volume tags.
• Individual users are free to choose their own backup mechanism.

On several occasions, Bob was contacted by users because they were not able to find recent backups to restore their applications. After manual investigation, Bob identified the following causes:

• Team A accidentally deleted a tag on volume 1, and their backup tool stopped backing up the volume.
• Team B modified their tag on volume 7, and their Amazon Data Lifecycle Manager custom policies stopped backing up the volume.
• Team C’s custom scripts had a software defect and stopped creating snapshots.
• Individual users assumed that Team B’s policies were set to protect all volumes with tag (production:environment) and thus tagged their volumes in that way. However, Team B’s policies were actually targeting volumes with a different tag (prod:environment). This meant individual users’ volumes were not backed up at all.

Solution overview

Both Alice and Bob can use Amazon Data Lifecycle Manager default policies to help them meet their needs:

Scenario 1: A single user wants a simple way to backup all resources

Alice can simply enable Amazon Data Lifecycle Manager default policies with a few clicks directly from the Amazon EC2 dashboard (Step 1 in the walkthrough). Alice can also monitor all EBS volumes in her account to make sure they all have backups created in the past day (Step 6).

Scenario 2: The storage administrator wants to make sure all critical workloads are backed up

Bob can easily enable the Amazon Data Lifecycle Manager default policy to make sure all critical applications used by the various teams are backed up on a daily basis. Although some users within the account set their own backup mechanisms to create backups every hour, Bob can use default policies as ‘catch-all’ backup protection so that important volumes at least have backups every day. When configuring his default policy (Step 4), Bob configures exclusion parameters so that only volumes running critical applications are targeted by the default policy.

Customers can use exclusion parameters to define attributes of EBS volumes and EC2 instances that they would like to exclude from being periodically backed up by the default policy. If a resource matches at least one of the exclusion parameters, then the default policy does not create backups of that resource.

Based on the requirements of the various teams and users, Bob’s default policy should exclude all boot volumes and non-io2 volume types (gp3, gp2, io1, sc1, st1, standard/magnetic). They should also exclude any volumes used for testing purposes when marked with a tag (temp:storage). Once they create the default policy, then all volumes running critical workloads for all teams and users are targeted by the default policy (coverage shown in the following sections). If each team’s backup mechanisms are working as planned to create hourly backups of their volumes, then the default policy does not create any additional snapshots. However, if any issues arise and a volume does not have a snapshot from the past day, then the default policy steps in to create a snapshot for that volume.

Prerequisites

The walkthrough assumes that you have EC2 instances and EBS volumes in your account that need to be backed up regularly. You do not need to apply tags to those resources in order for the default policies to create resources.

Walkthrough

There are several ways to create and manage Amazon Data Lifecycle Manager default policies. We have also integrated Amazon Data Lifecycle Manager default and custom policies with the Amazon EC2 console so customers can see whether they are in effect with launching new EC2 instances and creating new EBS volumes. In this post, we go through all the touchpoints of default policies:

1. Monitor default policies using the Amazon EC2 dashboard
2. Create and configure default policy settings
3. Differentiating between default policies and custom policies
4. Check default policy configuration while creating EBS volumes
5. Check default policy configuration while launching EC2 instances
6. Monitor EBS volumes and recent snapshot backups

Step 1: Monitor default policies using EC2 dashboard

1. Navigate to the Amazon EC2 dashboard and proceed to Data protection and security under Account Attributes.

2. In the Data protection and security tab, you can monitor whether Data Lifecycle Manager default policies is enabled for EBS Snapshots and/or EBS-backed AMIs. Select Create policy corresponding to the resource that you want to create.

If either/both have been created for the account in this AWS Region, then the dashboard also shows that policy’s creation frequency and retention period.

Step 2: Create and configure default policy settings

Once you have selected Create policy from the EC2 dashboard, you will be taken to the Data Lifecycle Manager policy creation page.

1. You can start configuring default policies by providing a Policy description and selecting an AWS Identity and Access Management (IAM) If you are unsure of which role to use, then we recommend choosing Default role. Otherwise, if you choose another role which does not have the correct IAM permissions, then the policy will move to the ERROR state.

2. Then you must set the Creation frequency, Retention period, as well as any Exclusion parameters.

Creation frequency defines the period whereby all EBS volumes targeted by the policy have snapshot backups. If there are other backup mechanisms creating snapshots at the same frequency (or more frequently), then your default policy will not create snapshots for those volumes. However, if other mechanisms are creating snapshots less frequently, or an issue arises which causes other mechanisms to stop creating snapshots for those targeted volumes, then default policy steps in and creates snapshots to meet the creation frequency criterion.

The Retention period defines how long snapshots should be retained once they have been created by the default policy.

You should define Exclusion parameters to set attributes for volumes that are not be targeted by default policies.

Based on Bob’s use case in the Solution section, the default policy does not create snapshots of:

• • Boot volumes
  • Volume types: gp3, gp2, io1, sc1, st1, standard/magnetic
  • Volumes with tag: (temp:storage)

If a team or user creates a volume that does not meet the preceding exclusion criterion, then the default policy targets that volume to make sure it has daily backups.

3. Next, you can set Advanced settings for the default policy. In this case, we have enabled Extend deletion to the last snapshot. If you do not enable this parameter, then by default the Amazon Data Lifecycle Manager default policy does not delete the last snapshot once the source volume has been deleted. It also does not delete snapshots created by the default policy after the policy has been disabled/deleted, or after it moves to error state. However, if you want the policy to continue deleting the final snapshot (based on the policy’s retention period) after the target volume has been deleted and after the policy is deleted/disabled or moves to ERROR state, then you should enable Extend deletion to the last snapshot.

4. Once you’re happy with the configuration, select Create default policy, confirm the configuration, and you have created the default policy! You can follow the same step to also create your default policy for EBS-backed AMIs, which target all EC2 instances in your account/Region, excluding those defined in the Exclusion parameters of the policy.

5. Customers can also easily create the same default policy using a single AWS Command Line Interface (AWS CLI)

$ aws dlm create-lifecycle-policy \
--state ENABLED \
--description "EBS snapshot policy" \
--execution-role-arn "role_arn" \
--default-policy VOLUME \
--create-interval 1 \
--retain-interval 7 \
--extend-deletion \
--exclusions ExcludeBootVolumes=true, ExcludeTags=[{Key=temp,Value=storage}], ExcludeVolumeTypes="standard|gp2|gp3|io1| st1|sc1"

Step 3: Differentiating between default policies and custom policies

You can also create Amazon Data Lifecycle Manager default policies through the Amazon Data Lifecycle Manager page in the Amazon EC2 console.

1. Select Data Lifecycle Manager in the Amazon EC2 console.

2. Select the radio button for Default policy. Alternatively, you can continue to create a Custom policy that targets resources based on tags.

3. Select either EBS snapshot policy or EBS-backed AMI policy. Then, you can follow the instructions in Step 2 to configure the policy.

4. If you have already created a default policy of the same type in the same account/AWS Region, then you cannot create another default policy for that resource, and the tiles are greyed out. You can continue to View default policies and modify existing default policies as needed.

5. In the main Amazon Data Lifecycle Manager screen, you can also display, monitor, filter, and sort default policies.

Step 4: Check your default policy configuration while creating EBS volumes

You can now also check if your Amazon Data Lifecycle Manager default and custom policies back up your EBS volume when you create the volume using the Amazon EC2 console.

1. Select Volumes under Elastic Block Store in the Amazon EC2 console and Create volume.

2. Scroll to the bottom of the page and select the refresh icon in the Snapshot summary.

3. If a default policy for EBS Snapshots has been configured for this account/AWS Region, then you can see whether the policy targets and backs up the volume once it has been created. You can also see if any Amazon Data Lifecycle Manager custom policies target this volume (based on any added tags).

4. If you do not want the volume to be targeted by the default policy, you must make sure the volume attributes match at least one of the Exclusion parameters. You can select View exclusion parameters for additional information. In this example, once we add the tag (temp:storage) and select the refresh icon again, we can see that the default policy no longer targets this volume.

Step 5: Check the default policy configuration while launching EC2 instances

You can now also check if Amazon Data Lifecycle Manager default and custom policies back up your EC2 instance when you launch the instance in the Amazon EC2 console.

1. Select Instances in the Amazon EC2 console and Launch instances.

2. If your default policy for EBS-backed AMIs is set to exclude instances based on tags and you want to exclude the current volume, then you should add the tag here and make sure it applies to Instances.

3. Scroll down to the Configure storage section of the page and select the refresh icon.

4. If the default policy for EBS-backed AMIs has been configured for this account/AWS Region, then you can see whether the policy targets and backs up the instance once it has been created. You can also see if any Amazon Data Lifecycle Manager custom policies target this instance (based on any added tags).

Step 6: Monitor EBS volumes and recent snapshot backups

Regardless of whether you use Amazon Data Lifecycle Manager, you can use the EBS Volumes page in the Amazon EC2 console to see how many of your volumes have snapshots created in the past 24 hours.

1. Select Volumes in the Amazon EC2 console.

2. You can also use the Search bar to filter based on volume attributes. Once you select more than one Volume on the page, you can navigate to the Selection summary tab to view the number of selected volumes with recent snapshots.

If you do not see recent snapshots for one or more volumes running critical workloads, then creating a default policy quickly helps you bridge this gap.

Cleaning up

Clean up the snapshots created during the previous steps to make sure you do not incur storage charges. You can do so by navigating to the Snapshots screen, searching for all snapshots created by the policy, selecting all the snapshots, and then selecting Actions followed by Delete snapshot.

Similarly, you should delete the Amazon Data Lifecycle Manager default policy to make sure no future snapshots are created by the policy. You should first enable Extend deletion property in the default policy to make sure any snapshots already created by the policy are deleted once the policy has been deleted. Then, you can delete the policy by navigating to the Lifecycle Manager screen, selecting the policy, and then selecting Actions followed by Delete lifecycle policy.

Conclusion

In this post, we covered creating Amazon Data Lifecycle Manager default policies to help account owners of AWS resources easily ensure all volumes in their account were backed up regularly. Through the use of default policies, storage admins can meet the requirements of the teams and users who use the same account that they manage. Storage admins gain the assurance that all critical workloads in the account are regularly backed up, even if a team’s or individual’s own backup mechanism fails. Best of all, default policies do not create additional backups if recent backups already exist, thereby eliminating additional costs and management overhead due to duplicated backups.

As a final suggestion, we encourage you to try this on your own environments. You can also learn more about this feature by reading our technical documentation.

Thank you for reading this blog. If you have questions or suggestions, leave them in the comments section.