How Liberty Mutual uses AWS Transfer Family to manage financial data

Liberty Mutual is the sixth largest property and casualty insurer in the world based on 2019 gross written premium. We rank 77th on the Fortune 100 list of largest corporations in the US based on 2019 revenue, and as of December 31, 2019 we had $43.2 billion in annual consolidated revenue. As a part of the larger enterprise, our Global Retail Markets (GRM) business unit has $28 billion in net written premium and a workforce of 30,000 employees across 15 countries.

In early 2020, we completed the migration of our applications from physical data centers to AWS and our operations now span six AWS Regions. While we have many cloud-native applications, we found AWS Transfer Family capabilities, specifically AWS Transfer Family for SFTP, a useful and familiar means of integrating with existing and new partners to exchange financial and regulatory data from banks and other sources. Before migrating to the cloud, deploying an SFTP server took weeks or months to create and fully integrate with our core systems prior to sharing with our partners.

Since AWS Transfer Family runs on a managed platform, AWS does most of the heavy lifting. The consumer does not have any EC2 instances to manage or configure. The service works as a platform for consumers to provision their own AWS Transfer Family servers within the guidelines dictated by the platform.

In this blog post, we focus on how we enabled and used AWS Transfer Family through an end-to-end automation process and how it helped our business go live with AWS Transfer Family SFTP servers within minutes to hours. We also discuss how our journey started and transformed through our use of AWS Transfer Family, lessons learned, challenges, and significant benefits of using the service.

How our journey started and transformed with AWS Transfer Family

As part of the migration into the cloud, our workloads require SFTP to exchange financial and regulatory data and to integrate that data with the other core systems. Around the same time that we started our migration (April 2019), AWS released a new service called AWS Transfer for SFTP (later renamed to AWS Transfer Family when FTP and FTPS protocol support was added earlier this year). The AWS Transfer Family service supports Amazon S3 as its backend data store, which ensures high durability and security of our data. In addition, this backend is well integrated with the rest of the AWS ecosystem. When AWS Transfer for SFTP was released, it supported two endpoint types – PUBLIC and VPC_ENDPOINT. When we began exploring which endpoint type was best for us, our goal was to open the SFTP endpoint through the internet through a Network Load Balancer in the public subnet. The downside of this was that a Network Load Balancer does not support security groups.

Earlier this year, AWS developed the VPC endpoint type, not to be confused with VPC_ENDPOINT mentioned in the preceding paragraph. This endpoint type uses an Elastic IP address, which opens the AWS Transfer Family server endpoint URL to the internet. Access is controlled through security groups on the AWS Transfer Family VPC endpoint. When the AWS Transfer Family server is provisioned, AWS creates the VPC endpoint interface and attaches the VPC default security group to it. The default security group can then be replaced with one or more custom security groups to control access.

The following diagram is a depiction of AWS Transfer Family server enablement with endpoint type VPC:

We enabled AWS Transfer Family through a multi-phase model. In the first phase, we enabled it with the SERVICE_MANAGED identity provider. In the second phase, we are planning to enable the CUSTOM_MANAGED identity provider using API Gateway with Amazon Cognito user pools by federating the identities from Azure AD.

Key features of AWS Transfer Family

• Available in all AWS Regions
• Supports custom domain names
• Supports service-managed users and custom-managed users
• Open to the internet but incoming traffic is securely protected at the VPC endpoint interface through security groups
• Amazon S3 is the backend for SFTP – S3 is the most performant cloud object storage
• AWS Transfer Family can transfer to multiple S3 buckets through a single server
• The consumer has options to enable S3 Lifecycle policies for data in Amazon S3

Benefits of AWS Transfer Family

• Zero maintenance and administrative responsibility on consumer
• Can completely automate the deployment
• Meets all our global cyber security guidelines
• Amazon S3 is integrated with AWS services and/or consumer data centers
• AWS Transfer Family pricing is low when compared with running SFTP in a data center
• Meets all our disaster recovery (DR) resilience guidelines

Some interesting things we learned in the process

• SFTP users are not IAM identities (like IAM role or IAM User); they are more like logical objects. These logical objects are associated with a managed IAM policy as a scope down policy and an IAM role to control the user’s access to the S3 bucket.
• AWS Transfer Family supports scope down policies to restrict the SFTP users to the designated S3 bucket and home directory.
• AWS recently introduced a “restricted” option that enforces root directory scope where users are not able to see the S3 bucket or home directory names. This feature replaces the need for a scope down policy.
• ${transfer:HomeFolder} and ${transfer:HomeDirectory} are scope down policy reserved variables. The difference between the two is that HomeFolder includes only the home directory, whereas HomeDirectory includes the bucket name plus the home directory.
• When the endpoint type is VPC, AWS Transfer Family routes the traffic to the SFTP server through the VPC endpoint interface created using consumer public subnets.
• There is a 1:1 ratio between the VPC endpoint interface and an AWS Transfer Family server. This is different from Amazon API Gateway and AWS Storage Gateway, where the VPC endpoint interface forwards communications to these services.
• AWS Transfer Family server computing charges are $0.30 per hour from started state to terminated state. These charges apply even when the AWS Transfer Family server is in a stopped state.

Our solution

In this example, we show how we implemented the AWS Transfer Family solution using the VPC endpoint type with service-managed users. Our solution is composed of Jenkins pipeline jobs to deploy the AWS CloudFormation template and additional jobs with Groovy scripts to manage users in the SFTP server. We use separate AWS CloudFormation templates to deploy the AWS Transfer Family SFTP server, S3 bucket, and the user roles and policies.

Our S3 bucket AWS CloudFormation resource creates an S3 bucket with a lifecycle configuration that moves the documents to lower price modes of storage as the document ages.

SFTPS3Bucket:
  Type: AWS::S3::Bucket
  DeletionPolicy: Retain
  Properties:
    AccessControl: Private
    PublicAccessBlockConfiguration:
      BlockPublicAcls: true
      BlockPublicPolicy: true
      IgnorePublicAcls: true
      RestrictPublicBuckets: true
    BucketEncryption:
      ServerSideEncryptionConfiguration:
      - ServerSideEncryptionByDefault:
          SSEAlgorithm: AES256
    BucketName: !Ref S3BucketName
    LifecycleConfiguration:
      Rules:
        - Status: Enabled
          Id: IA_Rule
          Transitions:
            - StorageClass: STANDARD_IA
              TransitionInDays: 90 
        - Status: Enabled
          Id: Glacier_Rule
          Transitions:
            - StorageClass: GLACIER
              TransitionInDays: 365
        - Status: Enabled
          Id: Deep_archive  
          Transitions:
            - StorageClass: DEEP_ARCHIVE
              TransitionInDays: 720

The next AWS CloudFormation template we deploy is our user role and policy template. This template deploys the scope down policy and the user role with an inline policy. This gives the user access to perform get and put operations on their home directory in the S3 bucket. The Ref’s are string parameters passed into the template. The following are the resources from this template:

Resources:
  UserS3Policy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: !Ref UserS3PolicyName
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowListingOfUserFolder
            Effect: Allow
            Action:
              - s3:ListBucket
              - s3:GetBucketLocation
            Resource:
              - !Sub arn:aws:s3:::${S3BucketName}
          - Sid: HomeDirObjectAccess
            Effect: Allow
            Action:
              - s3:PutObject
              - s3:GetObject
              - s3:DeleteObjectVersion
              - s3:DeleteObject
              - s3:GetObjectVersion
            Resource:
              - !Sub arn:aws:s3:::${S3BucketName}/*
      Roles: 
        - !Ref UserS3Role

  UserScopeDownPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Ref ScopeDownPolicyName
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowListingOfUserFolder
            Effect: Allow
            Action:
              - s3:ListBucket
            Resource:
              - arn:aws:s3:::${transfer:HomeBucket}
            Condition:
              StringLike:
                S3:prefix:
                  - ${transfer:HomeFolder}/*
                  - ${transfer:HomeFolder}

          - Sid: HomeDirObjectAccess
            Effect: Allow
            Action:
              - s3:PutObject
              - s3:GetObject
              - s3:DeleteObjectVersion
              - s3:DeleteObject
              - s3:GetObjectVersion
            Resource:
              - arn:aws:s3:::${transfer:HomeDirectory}*

  UserS3Role:
    Type: AWS::IAM::Role    
    Properties:
      Description: Role that will be assigned to users giving them access to an S3 bucket
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: transfer.amazonaws.com
          Action: sts:AssumeRole
      
      RoleName: !Ref UserS3RoleName

Given the VPC and public subnet IDs, this AWS CloudFormation template creates the Elastic IP address, Amazon CloudWatch logging policy, and AWS Transfer Family SFTP server.

Resources:  
  CloudwatchPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Action:
              - 'logs:CreateLogStream'
              - 'logs:DescribeLogStreams'
              - 'logs:CreateLogGroup'
              - 'logs:PutLogEvents'
            Resource: '*'
      PolicyName: !Ref 'LoggingRolePolicy'
      Roles:
        - Ref: 'SFTPLoggingRole'

  SFTPLoggingRole:
    Type: AWS::IAM::Role    
    Properties:
      Description: Cloudwatch logging role for the SFTP server to assume when logging connection and transfer activity
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: 'Allow'
          Principal:
            Service: 'transfer.amazonaws.com'
          Action: 'sts:AssumeRole'      
      RoleName: !Ref 'LoggingRoleName'           

  ElasticIp:
    Type: AWS::EC2::EIP
    Properties:
      Domain: 'vpc'    

  SFTPServer:
    Type: AWS::Transfer::Server
    Properties:
      EndpointType: 'VPC'
      EndpointDetails:
        VpcId: !Ref 'VPC'
        SubnetIds:
          - !Ref 'PublicSubnet'
        AddressAllocationIds:
          - !GetAtt 'ElasticIp.AllocationId'  
      IdentityProviderType: 'SERVICE_MANAGED'
      LoggingRole:
        !GetAtt 'SFTPLoggingRole.Arn'

When the AWS Transfer Family server is created through this AWS CloudFormation template or through the AWS Management Console, the default security group is automatically attached to the VPC endpoint. Since we use Jenkins as our pipeline tool, we have written a script that we call the Jenkinsfile to replace the default security group with our custom security group. In this custom group, only IP addresses that need access to this server are contained within the CIDR blocks we have specified. We use the AWS Java SDK to achieve this replacement.

Adding users

With the service-managed AWS Transfer Family server created, we look to add users. Since our countries of operation are consumers of this service, they are ultimately responsible for adding users to their own AWS Transfer Family servers. We provided Jenkins pipeline utilities to make it easier to manage users in their servers.

The util_sftp_add_user script takes a single parameter. This parameter is a text file that provides the Jenkinsfile with the server ID, user name, public key, home directory, role ARN, and restrict directory path. For the purposes of this example, we only discuss a user who is restricted through a scope down policy.

stage('sftp add user') {

  try {
    println("Adding user ${userName} to server ${sftpServer}.")
      def user = sftpAddUser(
                 serverId:"${sftpServer}",
                 userName:"${userName}",
                 publicKey:"${publicKey}",
                 role:"${roleArn}",
                 scopeDownPolicyArn:"${scopeDownPolicyArn}",
                 homeDir:"${homeDirectory}")
      if (user.length() != 0) {
          println("Successfully Added user: ${user} to SFTP server:
                 ${sftpServer}")
      } else {
          error("Failed to create user: ${userName} in SFTP server:
                 ${sftpServer}")
      }
    } catch (Exception e) {
        error("Failed to create user: ${userName} in SFTP server:
      ${sftpServer}")
    }      
 }

The preceding Jenkinsfile code snippet uses a utility library written in Groovy. This groovy script uses the AWS Java SDK to add the user to the AWS Transfer Family server. Part of that script is shown here:

. . .

CreateUserRequest request = new CreateUserRequest()
     .withServerId(serverId)
     .withUserName(userName)
     .withSshPublicKeyBody(publicKey)
     .withRole(role)
     .withHomeDirectory(homeDir)
     .withPolicy(policyArn)               
     .withTags(tags)                                      

CreateUserResult result = client.createUser(request)
return result.getUserName() 

. . .

Conclusion

Stable, secure, and cost-effective solutions are the primary focus in everything that we do at Liberty Mutual. We often achieve these goals by using purpose-built, fully managed, and fully featured services from AWS, like AWS Transfer Family. AWS Transfer Family comes “ready-to-consume,” is easy to deploy, and provides countless advantages from an operational perspective.