Protecting encrypted Amazon RDS instances with cross-account and cross-Region backups

Organizations are looking for solutions to protect their valuable data against ransomware attacks, natural disasters, and operational errors. Many of these organizations operate in regulated industries and must maintain data long-term to meet compliance obligations and business continuity goals.

In AWS, customers can accomplish these goals by backing up mission-critical databases into centralized backup storage in a different AWS account and Region.

AWS Backup supports the automated copying of Amazon RDS database snapshots from one AWS account to another or copying the backup from one AWS Region to another when the Amazon RDS database is encrypted using AWS KMS customer managed keys.  However, if the Amazon RDS database is encrypted using the default RDS KMS AWS managed key (aws/rds), AWS Backup cannot automate the cross-account backup currently.

In this blog post, we walk through a solution using AWS Backup together with Amazon EventBridge and AWS Lambda to automate copying Amazon RDS database snapshots (encrypted with the default KMS AWS managed key, aws/rds) from one AWS account to another AWS account in a different AWS Region.

While we will use Amazon RDS as the resource type to be backed up in this blog, this solution also works with other AWS Backup supported resource types, where the resource type does not support full AWS Backup encryption management, such as Amazon EBS, Amazon EC2, Amazon Aurora, Amazon FSx, Amazon DocumentDB, and Amazon Neptune. For more information, refer to the AWS Backup Developer Guide.

Solution overview

AWS Backup can perform cross-account backup for Amazon RDS encrypted with AWS KMS-CMK (Customer managed keys) because the AWS KMS-CMK can be shared across accounts. However, for Amazon RDS encrypted with KMS AWS managed key such as the default encryption key for RDS (aws/rds), the backups cannot be copied across accounts as the default encryption key for RDS cannot be shared across accounts. Review the blog post “Create and share encrypted backups across accounts and Regions using AWS Backup” to learn more.

The following diagram illustrates the high-level architecture of the solution, which uses AWS Backup to perform a backup, a same-account backup copy to an intermediate vault, and then another copy to achieve the cross-account backup copy.

Figure 1: AWS Backup cross-account and cross-Region architecture diagram

The solution uses an AWS Backup plan to create scheduled backups of the Amazon RDS database to a source vault. The same backup plan defines a copy rule, which copies the backup to an intermediate vault in the same account, but in a different Region. The solution then uses EventBridge to capture the successful AWS Backup copy job completion and uses these events to trigger an AWS Lambda function. The AWS Lambda function then uses AWS Backup to perform another backup copy job to copy the backup from the intermediate vault to the destination vault that is in a different AWS account.

The RDS DB instance encrypted with the default RDS KMS AWS managed key in the production account (account A) is backed up by AWS Backup to a SourceVault (configured to use cmk-source-vault-account-a-region-a encryption), and the backup is then copied to the IntermediateVault (configured to use cmk-intermediate-vault-account-a-region-b encryption). Although the SourceVault is configured to use “cmk-source-vault-account-a-region-a” encryption, the Amazon RDS snapshot that is stored on the SourceVault is still encrypted with the default RDS KMS AWS managed key (aws/rds). The step of copying the snapshot from SourceVault to IntermediateVault will decrypt the snapshot and re-encrypt the snapshot with “cmk-intermediate-vault-account-a-region-b” in the IntermediateVault. The “cmk-intermediate-vault-account-a-region-b” customer managed key can then be shared to the central backup account to be used for the cross-account copy.

After the copy to the IntermediateVault is complete, an EvenBridge rule in the production account (account A, Region A) identifies the “copy job complete” event and forwards the event to the EventBridge in account A, Region B, which in turn triggers a Lambda function. The Lambda function will check that the appropriate tags exist on the recovery points and will request AWS Backup to copy the backup from the production account (account A, Region B, IntermediateVault) to the central backup account (account B, Region B, DestinationVault) to achieve the cross-account and cross-Region backup copy.

At a high level, the following steps are involved in setting up cross-account backups between accounts in your organization. In this post, we provide an AWS CloudFormation template that accomplishes the following steps.

1. Create AWS Key Management Service (AWS KMS) customer managed keys in the production and central backup account.
2. Create a SourceVault (Region A) and an IntermediateVault in account A (Region B). Create a DestinationVault in account B (Region B). And assign the previously created KMS keys to each vault.
3. Create EventBridge rule in the production account (Region A and Region B) to identify the event of AWS Backup copy completion and trigger the Lambda function.
4. Create a Lambda function in the production account (account A, Region B) to trigger AWS Backup to copy the snapshot to the central backup account (account B).

Prerequisites and walkthrough

The following are the prerequisites for implementing the solution:

• An AWS Identity and Access Management (IAM) account with administrator privileges for the services used in this solution.
• An existing organization structure configured in AWS Organizations. And enabling the cross account feature of AWS Backup.
• Opting in for backing up the appropriate services, Amazon RDS in this case, in each Region that you want to use AWS Backup to manage backups.
• Create an Amazon RDS Follow the documentation to create an Amazon RDS database instance to be used as the Backup resource.
• Download the CloudFormation templates.

Familiarity with the following services:

• AWS Backup
• AWS CloudFormation
• Amazon EventBridge
• Amazon RDS
• AWS KMS
• AWS Lambda

Take note of the following:

• For Amazon RDS databases encrypted with AWS KMS customer managed keys, AWS KMS customer managed keys can be shared across accounts, and AWS backup can perform a cross-account copy within the same Region.
• For the RDS resource type, when AWS Backup performs a copy job, a single copy job can either perform a cross-account copy or a cross-Region copy. A single copy job cannot perform both cross-account and cross-Region at the same time. To achieve a cross-Region and cross-account copy, you will have to perform two copy jobs.
• Although the source vault is configured to use a KMS customer managed key for encryption, when the Amazon RDS database is being backed up to the source vault, the Amazon RDS backup is still encrypted with the RDS default KMS AWS managed key (aws/rds). The encryption information for the recovery points can be found using the AWS Backup CLI.
• For the solution in this post, make sure to choose the same Region for the intermediate vault and the destination vault.
• In this post, we will be using the source Region as US East (N. Virginia) (Region A) and the destination region as US West (Oregon) (Region B). And we will call the production account (account A) and the central backup account (account B)
• We will use the Amazon RDS as the resource type to be backed up. However, this solution also works with other AWS Backup supported resource types where the resource type does not support full AWS Backup encryption management.

Sign in to the management account of AWS Organizations and verify the prerequisite of two additional AWS accounts, excluding the management account.

Figure 2: AWS Organizations

Sign in to each account (management, production, and central backup account) and to the AWS Backup console. Under My Account, choose settings. Validate that cross-account backups have been enabled in AWS Backup in the management account and check on the opt in for backing up Amazon RDS in each Region, where AWS Backup is to be used.

Figure 3 shows an example of AWS Backup setting in the management account.

Figure 3: Enable AWS Backup features in AWS Organizations

Have the following information ready and download the CloudFormation templates:

• AWS account ID of the central backup account
• AWS Organization ID

Select the source account, source Region, and intermediate Region. As well as the destination account and destination Region. Make sure the intermediate vault is in the same Region as the destination vault.

Deploy the CloudFormation stack in the production account and source Region (account A, Region A)

In this section, we create a CloudFormation stack in the production account where the Amazon RDS database resides. The stack creates the following resources:

• AWS Backup vault
• KMS key for vault encryption
• IAM policy and role for EventBridge and a rule to forward the “copy job completed” event to the EventBridge in account A, Region B

1. Sign in to AWS CloudFormation console on your production account, and change to the source Region, where the RDS resource is located.

Figure 4: Production account and source Region where RDS resource is located

2. On the Create Stack menu, choose With new resources.

Figure 5: Create stack with new resources

3. For Template source, select Upload a template file.

4. Choose Choose file and upload “aws_backup_rds_source_account_a_region_a.yml” from the files downloaded, then choose Next.

Figure 6: AWS CloudFormation account-a Region-a

5. Under Specify stack details, provide a value for each parameter listed in the following table.

Figure 7: AWS CloudFormation parameters account-a Region-a

6. Time to finish up. Choose Next, then choose Next again. Select the check box to acknowledge IAM resource creation, and choose Create stack.

7. When the CloudFormation stack status changes to CREATE_COMPLETE, navigate to the Resources tab and validate that all the resources have been created successfully.

Figure 8: AWS CloudFormation created resources account-a Region-a

Deploy the CloudFormation stack in the production account and intermediate Region (account A, Region B)

In this section, we deploy a CloudFormation stack in the same production account but in a different Region. Choose the Region that will be the same Region as the destination vault in the central backup account. The stack creates the following resources:

• AWS Backup vault
• KMS key for vault encryption
• EventBridge rule
• Lambda function

1. Sign in to the AWS CloudFormation console in the production account. Change to the intermediate Region, which is the Region of the final destination vault.

Figure 9: AWS Console account-a Region-b

2. On the Create Stack menu, choose With new resources.

3. For Template source, select Upload a template file.

4. Choose Choose file and upload “aws_backup_rds_intermediate_account_a_region_b.yml” from the files downloaded, then choose Next.

Figure 10: Create stack with new resources

Figure 11: AWS CloudFormation account-a Region-b

5. Under Specify stack details, provide a value for each parameter listed in the following table.

Figure 12: AWS CloudFormation parameters account-a Region-b

6. Choose Next, choose Next again. Select the check box to acknowledge IAM resource creation, and choose Create stack.

7. When the CloudFormation stack status changes to CREATE_COMPLETE, navigate to the Resources tab and validate that all the resources have been created successfully.

Figure 13: AWS CloudFormation created resources account-a Region-b

Deploy the CloudFormation stack in the central backup account and destination Region (account B, Region B)

In this part, we deploy a CloudFormation stack in the central backup account in the desired destination Region. Choose the Region that is the same as the intermediate vault. The stack creates the following resources:

• AWS Backup vault
• KMS key for vault encryption

1. Sign in to the AWS CloudFormation console in the central backup account. Change to the destination Region, which is the Region of the final destination vault.

Figure 14: AWS Console account-b Region-b

2. On the Create Stack menu, choose With new resources.

3. For Template source, select Upload a template file.

4. Choose Choose file and upload “aws_backup_rds_destination_account_b_region_b.yml” from the files downloaded, and then choose Next.

Figure 15: Create stack with new resources

Figure 16: AWS CloudFormation account-b Region-b

5. Under Specify stack details, provide a value for each parameter listed in the following table.

Figure 17: AWS CloudFormation parameters account-b Region-b

6. Choose Next, and choose Next again. Select the check box to acknowledge IAM resource creation, and choose Create stack.

7. When the CloudFormation stack status changes to CREATE_COMPLETE, navigate to the Resources tab and validate that all the resources have been created successfully.

Figure 18: AWS CloudFormation created resources account-b Region-b

Creating the backup plan and adding the RDS resource to the backup plan

In this step, we will create a backup plan and add the RDS resource to the backup plan to be protected. When the backup plan is created, we will have to add a tag to the recovery points with the key:value of “CopyToVault:<ARN_of_destination_vault>”

1. Sign in to the AWS Backup console in the production account. Change to the source Region, where the RDS resource and the source vault are located.

Figure 19: AWS Console account-a Region-a

2. Choose Create Backup Plan.

Figure 20: AWS Backup console account-a Region-a

3. Choose Build a New Plan, and give your backup plan a name: backup-plan-rds.

Figure 21: AWS Backup console create new backup plan account-a Region-a

4. Give your backup rule a name: backup-rule-rds-daily-1week.

5. Choose the backup vault: source-vault-account-a-region-a.

Configure the Backup frequency and retention policy

1. Backup frequency: Daily.

2. Backup window: Default.

3. Transition to cold storage: Never.

4. Retention Period: 1 Week.

Configure the Backup copy policy

1. Copy to destination: Region B, for example, “US West (Oregon).”

2. Copy to another account’s vault: Disabled.

3. Destination backup vault: intermediate vault in Region B, for example, “intermediate-vault-account-a-region-b.”

4. Expand Tags added to recovery points and add the following tag. The key is CopyToVault, and the value is the <ARN of the destination vault in the central backup account>.

• CopyToVault : <ARN-of-destination-vault-in-Central-Backup-account>

5. Select Create Plan.

Figure 22: AWS Backup console create backup rule account-a region-a

Assign Resources to Backup Plan

1. Go to the created Backup Plan and choose Assign Resources.

Figure 23: AWS Backup console assign resources account-a region-a

2. Give a resource assignment name: rds-default-kms-key.

3. Choose an IAM role: Default role.

Figure 24: AWS Backup console assign resources account-a region-a parameters

4. Choose Include specific resource types.

5. Choose Resource Type: RDS and select the database instance that was previously created.

6. Select Assign resources.

Figure 25: AWS Backup console assign resources account-a region-a parameters

Validate the backups

The backups of the RDS resource will be taken daily and stored in the source vault (account A, Region A). The backup will be copied to the intermediate vault (account A, Region B) and then copied to the destination vault (account B, Region B). Wait for the daily backup and copy to take place before using the validation steps below.

The following steps validate that the backups have been created in the source vault and copied to the intermediate vault and destination vault, respectively.

• Sign in to the console in your production account (account A) in Region A.
• On the AWS Backup console, choose Backup vaults.
• Choose the vault created by the CloudFormation stack.

The following screenshots show backups of the RDS resource that we added to the backup plan in the source vault (account-a, region-a).

Figure 26: Source vault (account-a, region-a)

Repeat the previous process to validate that the backup was copied to the intermediate vault in the production account in Region B.

Figure 27: Intermediate vault (account-a, region-b)

Also, validate that the backup was copied to the destination vault in the central account (account B) in Region B.

Figure 28: Destination vault (account-b, region-b)

Cleaning up

Remove all the components created by this solution and avoid future charges. For all 3 of the environments, follow these steps to delete the snapshots in all the backup vaults first. Then, delete the CloudFormation stacks.

• Sign in to the AWS console of the AWS account and choose the Region.
• Go into the AWS Backup console, choose Backup vaults and choose the vault created by the CloudFormation template.
• Select all the recovery points in the vault, and on the Actions menu, choose Delete.
• Confirm the choice to delete the snapshots by entering “delete” and choose Delete recovery points.
• Go to CloudFormation and delete the stack.
• Repeat for all 3 environments.

Conclusion

In this post, we demonstrated how to use AWS Backup to automate the creation of backups for Amazon RDS DB instance encrypted with the defaul KMS AWS managed key (aws/rds) and copying the backups. We demonstrated both cross-Region and cross-account copy of the Amazon RDS database backups to meet regulatory compliance requirements or as a means to protect against ransomware attacks.

Please provide any comments or questions in the comments section. To learn more about AWS Backup, refer to the AWS Backup Documentation.