Simplify archiving Amazon EBS Snapshots and monitor progress using a live Amazon CloudWatch dashboard

Data protection is top of mind for our customers, and having a data backup strategy is critical to ensure compliance, disaster recovery readiness, and business continuity. As customers experience exponential business growth, their data storage needs grow as well, and data retention can become very costly. In order to meet compliance requirements for data retention while still being able to easily restore data, customers can save significant storage costs by moving infrequently accessed data to a colder storage tier.

Amazon Elastic Block Store (EBS) customers use EBS Snapshots to capture a point-in-time copy of their EBS volumes for backup compliance and disaster recovery. With the introduction of EBS Snapshots Archive, a new storage tier for archival, customers can now optimize storage costs and save up to 75% with this low-cost, long-term storage for rarely-accessed snapshots that do not need frequent or fast retrieval. To take full advantage of this archival tier, we recommend leveraging Amazon Data Lifecycle Manager – a policy-based lifecycle management solution for EBS Snapshots and EBS-backed Amazon Machines Images (AMIs) available at no additional cost – to create custom policies that automate the archival and retention of EBS Snapshots. Customers that need to keep full, point-in-time snapshots for backup and compliance can now do so by creating “set-and-forget” policies instead of having to manage and maintain complex custom scripts that may lead to data loss because of human errors.

For snapshots not created by Data Lifecycle Manager, we will discuss a solution that enables you to identify and archive eligible snapshots based on your criteria, in addition to monitoring via a custom Amazon CloudWatch dashboard that uses an AWS CloudFormation template. The proposed solution will automatically select snapshots for archival based on the tags and their age in regular intervals. It will also control the rate of requests to avoid throttling. Finally, it will provide a graphical overview of archived snapshots and current progress using AWS CloudWatch Metrics.

Solution overview

This solution consists of creating a serverless tool to automate snapshot archival and provide monitoring via a AWS CloudWatch dashboard. Once the solution is successfully implemented, you can access a Snapshot Archive Dashboard inside the CloudWatch console that shows the total number of snapshots in the account from the selected region, the number of archives in progress and the total cost as reported in AWS Cost Explorer. You will gain the ability to track snapshot archival progress and monitor the concurrent in-progress snapshot archive requests.

The solution involves a serverless approach using AWS Lambda, Amazon EventBridge, Amazon Simple Queue Service (Amazon SQS), and Amazon CloudWatch. The following image maps out the high-level diagram of components that will be deployed by the AWS CloudFormation template provided in this blog post:

Walkthrough

In order to deploy the components in the above diagram, you will need to complete the following steps to create a custom CloudWatch dashboard:

1. Log into the AWS Management Console and go to CloudFormation service.
2. Create a new stack by launching a CloudFormation template.
3. Specify stack parameters: this step determines the criteria of eligibility for snapshots archival and other operational options of the tool.
4. Add tags to track resources in Cost Explorer: monitor costs associated with resources created by the stack. Please note that this is an optional step that allows users to review costs on the usage of the tool in oppose to costs associated with snapshots.
5. Acknowledge AWS CloudFormation and select “Create stack”: by doing this users allow CloudFormation to create IAM and custom resources. The IAM resources give permissions for the stack deployment and tool functions while CloudFormation custom resource copy Lambda packages from a central location.

Prerequisites

As mentioned in the previous section, the solution is built on a serverless approach using AWS Lambda, Amazon EventBridge, Amazon SQS, and Amazon CloudWatch. Therefore, you will need to set up an AWS account with sufficient permissions to create resources in the aforementioned services.

Step 1: Log into the AWS Management Console and go to the CloudFormation service in the AWS Console

1. Log into the AWS Management Console and select the appropriate region.
2. Prepare to deploy a new stack to create resources using the following link: Creating a stack in the AWS CloudFormation console.

Step 2: Create a new stack by launching a CloudFormation template

1. Launch the following CloudFormation template directly from the CloudFormation console by specifying this URL as the ‘Amazon S3 URL‘.

Step 3: Specify stack parameters

Specify the following parameters to determine snapshot eligibility for archival and the operational options of the tool:

1. ArchiveInterval: Time between archives. If there is any manually archived snapshot within the interval, the next automated snapshots archival will be pushed further by the number of days defined in the interval.
2. MinimumAge: The minimum number of days the snapshot must exist before being archived.
3. Tags: Used to only archive snapshots with a specific Tag Key and Value. The criteria for selecting snapshots eligible for archival is based on the snapshot age, and optionally, tag(s). Tags consist of Key/Value pairs. The following syntax is used to to specify a tag:

tag:tag_name,Values=tag_value

• • tag: mandatory prefix (including colon character)
  • tag_name: the “Key” for the tag
  • tag_value: the “Value” expected for the respective key

Please refer to the following resource as an example:

The screenshot above is represented by the following configuration:

tag:Environment,Values=Production

It is possible to extend the selection to resources with the same tag Key and different values. Following the example above, if you want to archive snapshots from Staging and Production environments you would use the following configuration:

tag:Environment,Values=Production,Staging

If snapshots from the same volume have different key names for tags, only those matching the time and tag criteria will be archived. If you are looking to select the exact snapshots to archive, we recommend you to create an additional tag for them.

4. ExecTime: The time in which the tool scans for new snapshots eligible for archival. Please note that the time is in the UTC time zone. For example, the tool will scan the existing standard snapshot at 04:00AM UTC by selecting “4” in this option.
5. LogLevel: The level of details provided in the logs.
6. LogRetention:The number of days that the logs related to toll execution will be retained.
7. DryRun: If enabled by selecting “true”, the tool will only simulate the archival request without actually changing the snapshot tier. This is useful for testing. The dry-run execution messages will be available on CloudWatch logs under log group with prefix: /aws/lambda/SnapshotArchiveHandler-<region-name>.

Step 4: Add tags to track resources in Cost Explorer

You can also add tags to the resources used by the tool to keep track of the costs in Cost Explorer. Note that the tags defined under Configure stack options have no effect in the tool functionality:

Step 5: Acknowledge AWS CloudFormation and select “Create stack”

1. Acknowledge AWS CloudFormation to create IAM resources and Auto Expand:

2. Once the stack creation is completed, you can navigate to the Amazon CloudWatch console and select Dashboard, then select the created “SnapshotArchiveDashboard.”

You can see the following details in your dashboard:

• • Period: The time period in which data points will be selected and consolidated on the dashboard widgets. A change in the time period will alter the only monthly costs in the dashboard.

The remaining widgets reflect the current state and won’t change if a different period is selected:

• • Snapshots distribution: Graphical representation of EBS snapshots usage in number of resources per tier.
  • Archives in progress: Number of EBS snapshots being archived at the moment.
  • Queue size: Number of EBS snapshots waiting to be archived.

Cleaning up

To avoid incurring future charges for resources created in this blog post, delete the stack. Please note this will not delete any existing standard or archived snapshots.

Conclusion

In this blog post, we presented a solution for automating the archival of EBS Snapshots and implementing a custom CloudWatch dashboard to monitor progress. With the help of this live dashboard, you can track snapshot archives in the Region per account and look at the costs estimated by AWS Cost Explorer. This is important because you can now better plan your backup strategy by having the ability to track archival progress, monitor concurrent in-progress snapshot archive limit per account, and keep track of costs. With increased visibility and automatic archival you can reduce the risk of non-compliance, by moving snapshots to the archive tier without exceeding the API limits for archival requests.

Thank you for reading this blog. If you have any comments or questions, please leave them in the comment section below.