Automating shadow copies configuration on Amazon FSx for Windows File Server

We often hear from customers that they love the self-service, fine-grained file restore capability of the shadow copies feature on Amazon FSx for Windows File Server Filer Server (Amazon FSx). Our customers also relay to us the convenience of being able to schedule shadow copies on file systems using PowerShell commands. However, enabling Windows shadow copies on Amazon FSx can be a manual process for customers who use services such as AWS CloudFormation to automate their AWS infrastructure.

In this blog post, we cover using AWS CloudFormation to automate the configuration and scheduling of shadow copies. This can help lessen the administrative overhead if you have multiple Amazon FSx file systems to deploy, provide a repeatable process, and improve your security posture.

Comparing shadow copies and backups

Beyond automatically replicating your file system’s data to ensure high durability, Amazon FSx provides you two options to further protect the data stored on your file systems: Windows shadow copies enable your users to easily undo file changes and compare file versions by restoring files to previous versions, and Amazon FSx backups support your backup retention and compliance needs within Amazon FSx. The following chart compares shadow copies and backups.

Solution overview

If you prefer to start with deploying the solution first, skip to the “Deploying the solution” subsection for the necessary steps to do so. If you would like to follow along with the full tutorial, we go over the following topics:

• AWS Systems Manager Automation simplifies common maintenance and deployment tasks of Amazon EC2 instances and other AWS resources.
• AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. You use it to provision all the resources described in this solution and create a custom resource. An AWS CloudFormation custom resource enables you to write custom provisioning logic in templates that runs anytime you create, update, or delete a stack. They work by firing a webhook while your template is in processing, and your handler can then receive this webhook and run logic accordingly.
• AWS Lambda is the compute resource you use to execute the report generation code without provisioning or managing servers.
• Amazon FSx CLI for remote management on PowerShell enables file system administration for users in the file system administrators group.

Prerequisites

This solution assumes that you have already deployed the following components in your AWS account:

• Amazon Virtual Private Cloud (Amazon VPC) is the virtual private network where you deploy the compute resources.
• AWS Managed Microsoft AD provides the directory that you join the Amazon FSx for Windows File Server to. The Active Directory Domain Services on AWS Quick Starts can help you deploy AWS Directory Service in your environment.
• You must install and configure the AWS Command Line Interface (AWS CLI) on your machine.

Solution Architecture

Figure 1: Solution overview

Figure 1 depicts AWS CloudFormation invoking a custom resource AWS Lambda function. This function starts the execution of a Systems Manager automation command that runs on an EC2 instance. The EC2 instance is within a security group that allows communication between the instance and the Amazon FSx file server.

The Amazon FSx CLI commands executed configure the number of storage shadow copies your file server can consume. These commands also set the times and intervals, which can be daily, weekly, and monthly, that shadow copies are taken.

Tutorial

In the following sections, we break down the AWS CloudFormation template that deploys the solution. We start by taking a closer look at each of the components, followed by steps on deploying the solution, and end with how to clean up created resources to stop accruing charges. The steps are as follows:

1. AWS CloudFormation template parameters
2. Custom AWS Lambda resource
   1. Shadow copy schedule
   2. Custom resource AWS Lambda function
   3. Lambda execution role policy
3. Command execution
   1. Management EC2 instance
   2. AWS Systems Manager Automation
4. Solution deployment
5. Resource cleanup

AWS CloudFormation template parameters

To reference the prerequisite resources (for example, an Active Directory or subnet), multiple parameters have been defined in the AWS CloudFormation template. The following snippet provides all of the parameters along with descriptions for each. One item of note is that we use AWS Systems Manager Parameter Store public parameter to query for the latest Windows Server 2019 AMI ID. This eliminates the need of manually providing the latest AMI ID as either a parameter value or hardcoding it within the CloudFormation template. To learn more about this technique, read how to query for the latest Windows AMI using Systems Manager Parameter Store.

Parameters:
  LatestWindowsAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-windows-latest/Windows_Server-2019-English-Full-Base
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: Enter the security group ID to attach to the FSx for Windows file system
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: Enter the subnet ID that the FSx for Windows file system should be placed in
  ActiveDirectoryId:
    Type: String
    Description: Enter the Active Directory ID that the FSx for Windows file system will be associated with
  ActiveDirectoryName:
    Type: String
    Description: Enter the Active Directory ID that the FSx for Windows file system will be associated with
  ActiveDirectoryDnsAddresses:
    Type: List<String>
    Description: Enter the IP Address of the Active Directory DNS servers
  ManagementInstanceAvailabilityZoneName:
    Type: AWS::EC2::AvailabilityZone::Name
    Description: Enter the name of the Availability Zone that EC2 instance that will be used to manage the FSx for Windows file system will be placed in
  ActiveDirectoryCredentialsParameterName:
    Type: String
    Description: Enter the name of the Systems Manager Parameter Store parameter name containing the Active Directory credentials JSON string

Now that we know what parameters are needed, let’s dive a bit deeper into the other resources provisioned.

Shadow copy schedule

The AWS CloudFormation template creates an Amazon FSx File Server for Windows and a security group that allows inbound traffic for both file server access and management of the Amazon FSx file server. AWS CloudFormation then adds the file server to the Active Directory referenced in the parameter. The focus of this blog post and the resource that starts the process is an AWS Lambda-backed custom CloudFormation resource. Let us look at its declaration:

FSxShadowCopies:
  Type: Custom::FSxShadowCopies
  Properties:
    ServiceToken: !GetAtt SsmInvocationLambdaFunction.Arn
    fsxFileSystemId: !Ref TestFileSystem
    storageCapacity: "16%"
    scheduleExpressions: 
      - "-weekly -DaysOfWeek Monday,Tuesday,Wednesday,Thursday,Friday -at 06:00"
      - "-weekly -DaysOfWeek Monday,Tuesday,Wednesday,Thursday,Friday -at 18:00"
    instanceId: !Ref RemotePowerShellExecutionInstance
    ssmDocName: !Ref RunPowerShellInvokeCommandDocument
    activeDirectoryCredentialsParameterName: !Ref ActiveDirectoryCredentialsParameterName

The properties highlighted in purple are those that enable you to configure shadow copies. They follow the same syntax as the Set-FSxShadowStorage and New-ScheduledTaskTrigger PowerShell cmdlets, giving you the same experience as if you were executing the Amazon FSx CLI commands yourself. In the next section, let us look at the AWS Lambda function executed when AWS CloudFormation processes this resource.

Custom resource AWS Lambda function

The first step in the process is to have the AWS CloudFormation template execute the AWS Lambda function that executes the AWS Systems Manager Automation. To do this, we use an AWS Lambda-backed custom resource. In this section, we cover the declaration of the CloudFormation custom resource, the AWS Lambda function, and the IAM role that provides the Lambda function the necessary permissions.

First, we go over the portion of the AWS CloudFormation template that creates the AWS Lambda-backed custom resource:

ssmClient = boto3.client('ssm')

def handler(event, context):
    fsxClient = boto3.client('fsx')
    fileSystemId = event["ResourceProperties"]["fsxFileSystemId"]
    fileSystem = fsxClient.describe_file_systems(FileSystemIds = [fileSystemId])
    fsxRemotePowershellEndpoint = fileSystem["FileSystems"][0]["WindowsConfiguration"]["RemoteAdministrationEndpoint"]
    instanceIds = [event["ResourceProperties"]["instanceId"]]
    activeDirectoryCredentialsParameterName = event["ResourceProperties"]["activeDirectoryCredentialsParameterName"]
    try:
        eventType = event["RequestType"]
        # if we get a delete from CloudFormation, set the FSx schedule and storage to 0 (which is what Default is)
        if eventType == 'Delete':
            removeShadowStorage = 'True'
            scheduleExpressions = ''
        # if we get a Create from CloudFormation, set the FSx schedule and storage
        elif eventType == 'Create' or eventType == 'Update':
            removeShadowStorage = 'False'
            scheduleExpressions = '|'.join(event["ResourceProperties"].get("scheduleExpressions", []))
        result = ssmClient.send_command(
            DocumentName=event["ResourceProperties"]["ssmDocName"],
            InstanceIds=instanceIds,
            Parameters={"fsxRemoteEndpoint": [ fsxRemotePowershellEndpoint ], "removeShadowStorage": [removeShadowStorage], "storageCapacity": [event["ResourceProperties"]["storageCapacity"]], "scheduleExpressions": [scheduleExpressions], "activeDirectoryCredentialsParameterName": [activeDirectoryCredentialsParameterName]}
        )
        responseData = {'Result': result["Command"]["StatusDetails"], 'Version': ''}
        responseStatus = 'SUCCESS'  
    except Exception as e:
        responseStatus = 'FAILED'
        responseData = { 'Failure': f'Unable to fulfill {eventType} request. Check Lambda\'s CloudWatch log for details.' }

You can see that the AWS Lambda function creates a boto3 client, and then starts an AWS Systems Manager (SSM) Automation document passing in our desired parameters based off the event that AWS CloudFormation sends. If the function receives eventType == 'Delete' (that is, the stack is being deleted), then the AWS Lambda function calls the SSM Automation document with the variable removeShadowStorage equal to 'True'. This tells the SSM Automation document to remove the shadow storage. If the Lambda function receives eventType == 'Update'or ‘Create’ then the Lambda function calls our SSM Automation document with the parameters ‘storageCapacity’ and ‘scheduleExpression’. You need these parameters to create or update the Amazon FSx schedule and storage capacity. It is important to note that the Lambda function asynchronously invokes the SSM document. If you want to look at the execution logs, you must go to SSM’s run command history to see them.

Lambda execution role policy

Next, let us see what permissions our AWS CloudFormation templates gives to our function. We are using the AWS managed role called AWSLambdaBasicExecutionRole and then providing an additional two policies. The first enables our AWS Lambda function to start an SSM Automation execution. The second allows our function to query for the Amazon FSx file server’s Windows remote PowerShell endpoint.

LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      Path: "/"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      Policies:
        - PolicyName: "AllowSsmExecution"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action: "ssm:SendCommand"
                Resource: 
                  - !Sub "arn:${AWS::Partition}:ssm:*:${AWS::AccountId}:document/*"
                  - !Sub "arn:${AWS::Partition}:ec2:*:${AWS::AccountId}:instance/${RemotePowerShellExecutionInstance}"
        - PolicyName: "AllowDescribeFSxFileSystem"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action: "fsx:DescribeFileSystems"
                Resource: "*"

It is important to note that in order to execute the necessary Amazon FSx CLI commands, we need an EC2 instance that has network connectivity to the Amazon FSx file server. Let’s do that next.

Management Amazon EC2 instance

To execute the Amazon FSx CLI for remote management commands, the CloudFormation template creates a Windows compute instance that has network connectivity with the file server. You also place it in the FSx file server’s security group. The template then joins this EC2 instance to the Active Directoy by creating an association to the AWS-JoinDirectoryServiceDomain Systems Manager document.

AWS Systems Manager Automation

Now that we have created the custom AWS Lambda resource to run our SSM document, we must create the SSM document itself. To do this, we have our AWS CloudFormation template create a Systems Manager document that runs the necessary PowerShell code to enable the Amazon FSx schedule and storage capacity. The commands to manage shadow copies documented in the Amazon FSx for Windows File Server user guide. The following CloudFormation snippet creates the Systems Manager document that our custom AWS Lambda resource calls. Note that the snippet is an SSM command document as opposed to an Automation document.

    RunPowerShellInvokeCommandDocument:
    Type: AWS::SSM::Document
    Properties:
      DocumentType: Command
      Content:
        assumeRole: !GetAtt SsmExecutionRole.Arn
        description: Run a PowerShell Invoke-Command to start a psession and set the FSx Copy Schedule
        schemaVersion: "2.2"
        parameters:
          removeShadowStorage:
            type: "String"
          storageCapacity:
            type: "String"
          scheduleExpressions:
            type: "String"
          fsxRemoteEndpoint:
            type: String
            description: (Required) The Windows Remote PowerShell Endpoint for your FSx cluster
          activeDirectoryCredentialsParameterName:
            type: String
            description: (Required) The Systems Manager Parameter Store parameter name containing the Active Directory credentials JSON string
        mainSteps:
          - name: RunPowershellInvokeCommand
            action: "aws:runPowerShellScript"
            precondition:
              StringEquals:
                - "platformType"
                - "Windows"
            inputs:
              runCommand:
              - "$param = (Get-SSMParameter -Name '{{activeDirectoryCredentialsParameterName}}' -WithDecryption $True).Value | ConvertFrom-Json"
              - "if (${{removeShadowStorage}}) {"
              - "  Invoke-Command -ComputerName {{fsxRemoteEndpoint}} -configurationname fsxremoteadmin -credential (New-Object System.Management.Automation.PSCredential $param.username,(ConvertTo-SecureString $param.password -AsPlainText -Force)) -ScriptBlock { Remove-FsxShadowStorage -Confirm:$False }"
              - "}"
              - "else {"
              - "  Invoke-Command -ComputerName {{fsxRemoteEndpoint}} -configurationname fsxremoteadmin -credential (New-Object System.Management.Automation.PSCredential $param.username,(ConvertTo-SecureString $param.password -AsPlainText -Force)) -ScriptBlock { Set-FsxShadowStorage -MaxSize '{{storageCapacity}}' }"
              - "  [CimInstance[]]$triggers = @()"
              - "  foreach ($triggerExpression in \"{{scheduleExpressions}}\".Split('|')) {"
              - "    $scriptBlockExpression = \"param([CimInstance[]]$triggers) New-ScheduledTaskTrigger $triggerExpression\""
              - "    $scriptBlock = [ScriptBlock]::Create($scriptBlockExpression)"
              - "    $triggers += Invoke-Command -ScriptBlock $scriptBlock -ArgumentList $triggers"
              - "  }"
              - "  Invoke-Command -ComputerName {{fsxRemoteEndpoint}} -configurationname fsxremoteadmin -credential (New-Object System.Management.Automation.PSCredential $param.username,(ConvertTo-SecureString $param.password -AsPlainText -Force)) -ScriptBlock { Set-FsxShadowCopySchedule -scheduledtasktriggers $Using:triggers -Confirm:$False }"
              - "}"

Deploying the solution

The first thing that you must do is create the AWS Systems Manager Parameter Store parameter that you use to store the user name and password of a member of the file server administrators group. You must do this first because you use this parameter in later steps. If you are using AWS Managed Microsoft AD, that group is AWS Delegated FSx Administrators.

Execute the following AWS CLI command, substituting the placeholders to create the parameter.

aws ssm put-parameter --name "/fsx-sample/admin-credentials" --type "SecureString" --value '{ "username": "<User Name>", "password": "<Password>" }'

Make sure that you escape any backslashes or double-quotes in the user name and password as this is a JSON string. For example, as depicted below, instead of using CORP\Admin for the user name, use CORP\\Admin.

aws ssm put-parameter --name "/fsx-sample/admin-credentials" --type "SecureString" --value '{ "username": "CORP\\Admin", "password": "<Password>" }'

Next, download the complete template. Once you have created the file, you can deploy the solution by running the following AWS CLI command in the same directory that you downloaded the file to, making sure to replace the placeholders with values for your environment.

aws cloudformation deploy --stack-name fsx-shadow-copy \
--template-file fsx-shadow-copy.yml \
--parameter-overrides \
VpcId=<Your VPC ID> \
SubnetId=<Subnet ID> \
ActiveDirectoryId=<Directory Name> \
ActiveDirectoryName=<Active Directory Name> \
ActiveDirectoryDnsAddresses=<Active Directory DNS IP Address> \
ManagementInstanceAvailabilityZoneName=<Availability Zone> \
ActiveDirectoryCredentialsParameterName="/fsx-sample/admin-credentials" \ 
--capabilities CAPABILITY_IAM

Verifying shadow copy settings

To verify that you have enabled your shadow copy settings, you can open up the console, then:

1. Click the Services button in the top left of the console
2. Navigate to Systems Manager.
3. Select Run Command on the left-hand side of the screen (under Instances & Nodes).
4. Near the top of your screen, you there are two tabs title Commands and Command history – select Command history.
5. Select the command ID that has your stack name in the document name (for example, fsx-shadow-copy- RunPowerShellInvokeCommandDocument-ZGUTO3BKVKK1).
6. Select the Instance ID that is in the ‘Targets and outputs’ section of the page.
7. Expand the Output section (Titled Step 1 – Output).
8. Once doing so, you should see output with your Shadow Copy settings that looks like the following:

Cleaning up

To avoid incurring future charges, you can use the following AWS CLI command to delete the AWS CloudFormation stack created earlier and its corresponding resources.

aws cloudformation delete-stack --stack-name fsx-shadow-copy

Conclusion

In this blog post, we covered using services such as AWS Systems Manager, AWS CloudFormation, and AWS Lambda, to automate the administrative task of setting up shadow copies on a Amazon FSx file server. This same technique can be applied to other Amazon FSx remote management commands (for example, storage quotas and data deduplication), letting administrators focus on more value adding activities. The ease of automation enables you to save time as you leverage shadow copies for Amazon FSx, removing administrative overhead while providing additional backup protection in the form of moment-in-time snapshots.

Amazon FSx for Windows File Server offers fully managed Microsoft Windows file servers, backed by a fully native Windows File Server. Enabling shadow copies on Amazon FSx can enable your users to easily view and restore individual files or folders from an earlier snapshot in Windows File Explorer.

To learn more about Amazon FSx for Windows File Server see the following:

• Amazon FSx documentation
• Amazon FSx shadow copies video tutorial
• Amazon FSx shadow copies

To learn more about AWS Systems Manager, AWS CloudFormation, and AWS Lambda, see:

• AWS Systems Manager
• AWS CloudFormation
• AWS Lambda

Thanks for reading this blog post! If you have any comments or questions, please don’t hesitate to leave them in the comments section.