ZS Associates enhances backup efficiency with AWS Backup

ZS Associates is a professional services firm that works alongside companies to help develop and deliver products that drive customer value and company results. We at ZS leverage our deep industry expertise, alongside leading-edge analytics, technology, and strategy to create solutions that work in the real world. With more than 35 years of experience and 8,000-plus ZSers in more than 25 offices worldwide, we at ZS are passionately committed to helping companies and their customers thrive.

Background

ZS focuses on delivering data-backed services, and as a result, we depend heavily on robust backup and disaster recovery (DR) strategies. This is no easy feat for us, given our large-scale operations comprising multiple cloud applications hosted across more than 150+ AWS accounts. ZS runs many of its data applications and storage on AWS, leveraging services like Amazon S3, Amazon EFS, Amazon EC2, Amazon Redshift, Amazon RDS, and Amazon Aurora. We needed a data protection solution that not only manages our backups in a consistent manner, but also adheres to enterprise security and compliance standards.

In this blog post, we describe:

• How we deployed AWS Backup to centrally manage and automate data protection across AWS services.
• The benefits of this solution, including savings such as operational costs and staff-hours.

Challenges we faced before using AWS Backup

Before using AWS Backup, ZS relied on third-party backup tools; however, none of these tools met ZS’s comprehensive expectations. While these third-party tools supported backing up certain AWS services, ZS had to implement home-grown tooling to encompass AWS services that weren’t supported by the third-party vendor. ZS also had to build a custom governance framework that ensured a single pane of glass across our AWS landscape. Aligning with ZS’s information security policies required additional tooling that oversaw backup aging, retention, and replication across each applicable AWS service. Overall, this resulted in a compliant, but complex ecosystem that required ongoing oversight and enhancements.

After onboarding to AWS Backup

When AWS Backup launched in January 2019, ZS was one of the first customers to onboard. AWS Backup made it easy for ZS to centralize and automate our backup processes across our AWS accounts. This allowed ZS to have a consistent approach for backup and DR across numerous AWS services. AWS Backup simplified backup key management, lifecycle policy enforcement, organization-wide rollout, and governance. AWS Backup’s robust built-in data protection capabilities gave ZS additional assurance that our backups are secure from malicious or unintended deletions.

7 key benefits ZS experienced using AWS Backup

1. Uniform and centralized backup policy across multiple AWS services

With AWS Backup, we were able to apply a uniform policy across various AWS services. More importantly, we were able to manage policies centrally. We also experienced the following benefits:

Configuring backups for multiple services

Since AWS Backup supports many of the commonly used AWS services that we mentioned above, we were able to define and configure a uniform backup policy across various AWS services and accounts. It also allowed us to accommodate backup for cluster databases like Amazon Aurora. This capability enabled us to automate our compliance backup needs across many AWS services.

Managing and monitoring backups across accounts

Given that AWS Backup’s cross-account functionality integrates with AWS Organizations, it helped us to manage and monitor backup processes from a single management account. This also reduced the operational overhead of configuring and managing policies, plans, and vaults in each account.

Deploying and updating across accounts

We have been using the AWS CloudFormation StackSets to create and update the components of AWS Backup (such as backup vaults, backup plans, and backup IAM roles). Furthermore, using AWS Backup cross-account policies, we centrally manage the backup plans and rules from a management account along with deploying it to new accounts. This simplifies and reduces the effort for deploying and updating AWS Backup across accounts.

2. Eliminate the need for third-party tools and custom scripts

None of the third-party tools we used covered all the commonly used AWS services. As a result, we had to create additional custom scripts to backup/replicate in addition to create another set of separate scripts for cleanup. With AWS Backup, we no longer need such scripts, removing the significant overhead involved in maintaining them and keeping them updated with the changes in AWS API operations.

3. Eliminate repeated requests to increase snapshot limits for each account

AWS Backup also helped overcome some limitations like soft limits on:

• Snapshot counts per AWS account (service quota)
• Concurrent AMI cross-region replication counts
• Concurrent snapshot replication counts

These limitations added operational overhead and manual intervention that resulted in delays. AWS Backup handles these limits without any additional effort in requesting to either increase the limit or have multiple batch jobs to overcome them.

4. Eliminate the need to use AWS Data Pipeline for Amazon EFS backup

Before using AWS Backup, we had to maintain a separate process to back up Amazon EFS using AWS Data Pipeline. With AWS Backup, the process is simplified. We do not have to maintain the AWS Data Pipeline for it and we have a more consistent backup process in line with other AWS services. In addition, in AWS Backup, the encryption is applied at the vault level, further simplifying the encryption management for Amazon EFS.

5. Snapshots deletion protection

AWS Backup provides an extra layer of security by preventing accidental deletion of recovery points (backups) from the corresponding service console (shown in the following screenshot). Snapshots can only be deleted via the AWS Backup console or API operations with appropriate access roles. With this setup, we do not need additional explicit IAM roles and policies to safeguard against deleting the snapshots from unintended actions.

6. Facilitates better on-demand backup creation and management

Often, there is a need to create on-demand backups. We can create on-demand backups for services such as Amazon EC2, Amazon RDS, etc., using the AWS Management Console or API. However, once we have created these backups, we need a separate automated mechanism to manage their lifecycle as these backups can create additional costs with growth over time. There is also a risk of accidentally deleting snapshots, especially during the backup creation process.

We have used AWS Backup to improve the on-demand backup process through better backup lifecycle management processes and extra protection.

Better backup lifecycle management process

The AWS Backup API method StartBackupJob(**kwargs) enables us to specify the lifecycle for an on-demand backup. Using this method, we have automated our backup creation and retention management process, without which, we would need to rely on a separate cleanup process for the older snapshots.

Extra protection

AWS Backup provides the ability to keep all backups protected in vaults. This provides an additional layer of protection, compliance, and governance from accidental deletion.

7. Covered by AWS Support

AWS Backup is covered by AWS Support and is included under the AWS Enterprise Support as well.

Benefits looking forward: cold storage options

AWS Backup has started offering the option to move backups to low-cost cold storage after a support specified duration (currently, this option is only available for Amazon EFS). This option gives the advantage of saving cost on backups, especially when there is a need to retain backups for a longer period to fulfill compliance obligations. We are looking forward to AWS Backup extending this functionality to other services over time.

Setting up and managing AWS Backup across AWS accounts

The following section describes the process ZS followed for setting up and configuring AWS Backup in a multi-account setup using AWS Organizations and AWS CloudFormation StackSets.

Backup prerequisites created via AWS CloudFormation

We used the AWS CloudFormation StackSets to centralize the creation and further manage the prerequisites (the IAM role and the backup vaults) required for AWS Backup. The CloudFormation template (more details on this template shared as an example in following sections) is deployed from the management account and creates the IAM role and the required backup vaults in the child accounts that are part of the Organization.

IAM role

• This IAM role is used by AWS Backup for backup and restore operations, such as creating, restoring, or expiring backups. It is also used in backup plans for resource assignments and copying the backups to the secondary Region.

Backup vaults

• Backup vaults are required in primary and secondary Regions to store recovery points.

Using a comprehensive tagging strategy

Developing a comprehensive and automated tagging strategy helped us easily assign resources to backup plans. We provide an example in the “Resource assignments” section later in this blog post.

Automated backup configuration to newer accounts

We used service-managed CloudFormation StackSets and configured the necessary IAM permissions required to deploy the stack in the AWS Organizations’ accounts. The service-managed permissions helped in automating the AWS Backup deployment in the newer accounts. Whenever we add a new AWS Backup account to the OU (organizational unit), the backup CloudFormation template is automatically deployed in that account, creating the required backup vaults and an IAM role.

Using the Same CloudFormation template for primary and secondary AWS Regions

We used the same StackSets for both primary and secondary Regions. We used the CloudFormation mapping to obtain the required primary and secondary Regions, and the pseudo parameters and CloudFormation conditions to identify the Region during the runtime. This allowed us to use the same CloudFormation template in both primary and secondary Regions.

The following is a sample CloudFormation stack for illustration.

YAML

AWSTemplateFormatVersion: "2010-09-09"
Description: "Backup Plan template for backup and replication"

Mappings:
  RegionToRegionNameMap:
    us-xxx-x:
       "DisplaynameForRegionSpecificResources": "US-primary"
       "DisplaynameForGlobalResources": "US"
    us-xxx-x:
       "DisplaynameForRegionSpecificResources": "US-secondary"
       "DisplaynameForGlobalResources": "US"
    eu-xxx-x:
       "DisplaynameForRegionSpecificResources": "EU-primary"
       "DisplaynameForGlobalResources": "EU"
    eu-xxx-x:
       "DisplaynameForRegionSpecificResources": "EU-secondary"
       "DisplaynameForGlobalResources": "EU"
    ap-xxx-x:
       "DisplaynameForRegionSpecificResources": "AP-primary"
       "DisplaynameForGlobalResources": "AP"
    ap-xxx-x:
       "DisplaynameForRegionSpecificResources": "AP-secondary"
       "DisplaynameForGlobalResources": "AP"

Conditions

  IsPrimaryRegion: !Or [!Equals [!Ref 'AWS::Region', us-xxx-x], !Equals [!Ref 'AWS::Region', eu-xxx-x], !Equals [!Ref 'AWS::Region', ap-xxx-x]]

  IsSecondaryRegion: !Or [!Equals [!Ref 'AWS::Region', us-xxx-x], !Equals [!Ref 'AWS::Region', eu-xxx-x], !Equals [!Ref 'AWS::Region', ap-xxx-x]]

Resources

  BackupRuleRole:
    Type: AWS::IAM::Role
    Condition: IsPrimaryRegion
    Properties:
      RoleName:
        Fn::Join:
          - "-"
          - - !FindInMap [RegionToRegionNameMap, !Ref "AWS::Region", "DisplaynameForGlobalResources"]
            - backup_role

      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup
        - arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForRestores
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
              - backup.amazonaws.com
          Action: sts:AssumeRole         
      Path: "/"

  BackupVaultForCustomBackups:
    Type: "AWS::Backup::BackupVault"
    Properties:
      BackupVaultName:
        Fn::Join:
          - "-"
          - - !FindInMap [RegionToRegionNameMap, !Ref "AWS::Region", "DisplaynameForRegionSpecificResources"]
            - backup
            - vault01

Using AWS Backup cross-account management for creating and managing backup policies

We used the AWS Backup cross-account management to automatically apply backup plans across multiple accounts belonging to AWS Organizations from a single management account. It also allows us to create/update/delete the policy with all its key components. Another advantage of cross-account management is the extra layer of governance it provides on the backup policy. We cannot alter the backup policy from the respective accounts. This ensures we have consistent backups across accounts and avoid any unintentional deviation on an account level.

Illustration of setting up and using AWS Backup

For this example, we create a backup plan where we have a daily backup and replication of AWS resources based on tag values assigned to them.

We navigate to the AWS Backup console and enable the Backup policies and Cross-account monitoring features in the AWS Backup settings.

Then, we create a backup policy by selecting My organization, then Backup Policies.

Backup plan

The Backup plan details consist of the backup plan name and the Region to deploy the backups. Here we are deploying in our primary Region.

Backup rule

For our backup rule, we define the frequency, lifecycle, and copy/replication configuration of backups. Next, we are creating a backup rule with name “Production_Daily_Backup_Replication_Rule,” which has a daily backup frequency and backup window.

Lifecycle and copy configuration

In the lifecycle configuration, we defined the expiration for backups (we are currently not transitioning any backups to lower tiers as it is only supported for Amazon EFS). For copying the backups to a secondary Region, we can specify the destination Region and specify the backup vaults that we created using StackSets.

Resource assignments

Lastly, for our backup policy, we specify resources that will be backed up and replicated using a tag-based approach. We also need to specify the IAM role that we created using the StackSet. The tag values we used in this case are:

• Resource tag key: Environment
• Tag values: Production

The backup policy can be created via visual editor as shown in the previous screenshot or by using the following JSON template.

JSON

{
    "plans": {
        "Production_daily_Backup_Replicaiotn_plan": {
            "regions": {
                "@@assign": [
                    "us-xxx-x"
                ]
            },
            "rules": {
                "Production_daily_Backup_Replicaiotn_Rule": {
                    "schedule_expression": {
                        "@@assign": "cron(x x x x x x)"
                    },
                    "start_backup_window_minutes": {
                        "@@assign": "xx"
                    },
                    "complete_backup_window_minutes": {
                        "@@assign": "xxx"
                    },
                    "lifecycle": {
                        "delete_after_days": {
                            "@@assign": "xxx"
                        }
                    },
                    "target_backup_vault_name": {
                        "@@assign": "US-primary-Backup-vault01"
                    },
                    "copy_actions": {
                        "arn:aws:backup:us-xxx-x:$account:backup-vault:US-secondary-Backup-vault01": {
                            "lifecycle": {
                                "delete_after_days": {
                                    "@@assign": "xx"
                                }
                            },
                            "target_backup_vault_arn": {
                                "@@assign": "arn:aws:backup:us-xxx-x:$account:backup-vault:US-secondary-Backup-vault01"
                            }
                        }
                    }
                }
            },
            "selections": {
                "tags": {
                    "Production_resource_assignmnet": {
                        "iam_role_arn": {
                            "@@assign": "arn:aws:iam::$account:role/US-backup_role"
                        },
                        "tag_key": {
                            "@@assign": "Environment"
                        },
                        "tag_value": {
                            "@@assign": [
                                "Production"
                            ]
                        }
                    }
                }
            }
        }
    }
}

Assign targets

Next, we attach the backup policy to accounts or organizational units (OUs) within the organization. We do this by selecting the newly created backup policy and adding the required targets to the policy. We then deploy the backup policy in those accounts or OUs.

Summary

In this blog post, we described how we implemented AWS Backup at ZS, which helped centrally manage and automate backup deployments in a multi-account environment. We also shared seven key benefits that we experienced with AWS Backup while working on this project. As a result of adopting AWS Backup, ZS deprecated the use of custom-built scripts and processes. The transition has raised ZS’s operational efficiency and resulted in a yearly savings of $30,000 coupled with saving about 1,200 person-hours in backup operations. As a result, the AWS Backup service has significantly helped us minimize manual efforts and costs without sacrificing any of our backup compliance needs. Given the significant benefits we obtained, AWS Backup has become an essential component of our backup strategies. We look forward to AWS Backup’s support for the other AWS services and features.

Thanks for reading this blog post! If you have any questions or suggestions, please leave your feedback in the comments section.