AWS Storage Blog

Transferring files securely at the Foxtel Group using AWS Transfer Family and Amazon EFS

Foxtel Group

The Foxtel Group is Australia’s leading, next-generation subscription television company providing movies, entertainment, lifestyle, documentaries, news, and live sports through a range of streaming and broadcast services to over four million subscribers.

Enterprises such as the Foxtel Group rely upon secure file transfer solutions for the timely and secure transfer of data between themselves and external parties. They want to migrate from their existing on-premises file transfer platforms to AWS whilst ensuring a consistent experience and minimal disruption for existing clients.

Using AWS Transfer Family and Amazon Elastic File System (Amazon EFS) together provided an easy way for the Foxtel Group to make the transition to AWS. However, they faced a challenge replicating the NFSv4 access control lists (ACLs) that were used to restrict access to sensitive files and directories for both external SFTP users and internal hosts mounting the on-premises file system.

In this post, we walk through how AWS Partner Tata Consultancy Services (TCS) replaced the Foxtel Group’s on-premises SFTP platform with an AWS Transfer Family SFTP server backed by Amazon EFS storage.

We show how capabilities provided by AWS Transfer Family and Amazon EFS deliver an equivalent level of security to on-premises NFSv4 ACLs.

Solution overview

The following diagram shows the key components that we used to deliver an AWS Transfer Family SFTP server backed by Amazon EFS and make it available to hundreds of third parties over the internet without compromising security.

Architectural diagram illustrates secure file transfer via AWS Transfer Family and Amazon EFS. External users SFTP files to EFS via Transfer Family, EFS is accessed from EC2 hosts via EFS access points.

AWS Transfer Family is a fully managed FTP, FTPS, and SFTP service backed by either Amazon S3 or Amazon EFS. It frees customers from managing any server infrastructure or disrupting existing file transfer workflows.

Amazon EFS provides a simple, serverless, set-and-forget elastic file system for use with AWS Cloud services and on-premises resources. Amazon EFS elastically scales on demand from zero to petabytes with no disruptions, growing and shrinking automatically as you add and remove files, removing the need to provision and manage capacity. Amazon EFS works with all AWS compute services, including Amazon EC2 instances, AWS container services, and AWS Lambda.

Amazon EFS is a Regional service providing strong file system consistency across three Availability Zones, matching the AWS Transfer Family support for high availability across up to three Availability Zones.

We migrated the external SFTP users to AWS Transfer Family by adding them as service-managed Transfer Family users. If you want to use password authentication or an external identity provider rather than service-managed users, you should follow the approach documented on the AWS Storage Blog. We created the users with their on-premises POSIX user and group IDs and set their home directory attribute within Transfer Family to their directory on the Amazon EFS file system to constrain their access to the file system.

To manage application access to shared datasets, we used Amazon EFS Access Points. Access points provide application-specific entry points into an EFS file system and can enforce a particular POSIX user identity, including the user’s groups, for all file system requests that are made through the access point, irrespective of the identity of the client. Access points can also enforce a different root directory for the file system so that clients can only access data in the specified directory or its subdirectories.

At the Foxtel Group, the application hosts reside in different VPCs from the Amazon EFS file system mount targets, and connectivity between the VPCs was enabled using an AWS Transit Gateway. However, the approach will also work if you have a VPC peering connection between your VPCs or if the workloads run in the same VPC as the file system.

Solution walkthrough

We demonstrate the approach with a simplified example of two external parties transferring files via AWS Transfer Family to dedicated Amazon EFS directories and two internal applications consuming that data. The first application has read and write access to the directory used by the first external party, and the second application has read-only access to both directories.

The walkthrough consists of the following steps:

  1. Creating an Amazon EFS file system.
  2. Creating IAM roles for Amazon EFS access.
  3. Creating an AWS Transfer Family SFTP server and Amazon EFS directories.
  4. Adding SFTP users to AWS Transfer Family.
  5. Creating EFS Access Points to enable access for Amazon EC2 instances.
  6. Updating the Amazon EFS file system IAM policy to enable access points.
  7. Mounting Amazon EFS via access points from Amazon EC2 instances.

Step 1: Creating an Amazon EFS file system

Create an Amazon EFS file system as a location for our AWS Transfer Family SFTP server to store the files it receives. For full details on how to do this, refer to the blog post “Making it even simpler to get started with Amazon EFS.”

Set a file system IAM policy to lock down access by checking the boxes Prevent root access by default, Enforce read-only access by default, and Prevent anonymous access.

Also check Enforce in-transit encryption for all clients to enforce encryption in transit for NFS client access to Amazon EFS, ensuring that all traffic is encrypted using Transport Layer Security 1.2 (TLS) with an industry-standard AES-256 cipher.

Under File System Policy settings, the Policy options to Prevent root access by default, Enforce read-only access by default, Prevent anonymous access and Enforce in-transit encryption for all clients are selected.

The resulting file system policy allows no actions against the file system, but when a client connects to Amazon EFS, any identity-based IAM permissions associated with the client are evaluated along with the file system policy to determine the appropriate file system access permissions to grant.

Step 2: Creating IAM roles for Amazon EFS access

This step creates IAM roles required to administer the EFS file system, to upload files via Transfer Family, and to access the file system from EC2 instances.

To add an IAM role that provides administrative rights to EFS, navigate to the IAM console and choose Create role.

The Create Role button is located on the top right of the IAM console.

Select AWS service for Trusted entity type and select Transfer from the dropdown under Use case, Use cases for other AWS services.

The Select trusted entity section displays a Trusted entity type set to AWS service and Transfer selected from the dropdown for the Use cases for other AWS Services field.

Search for and select the AmazonElasticFileSystemClientFullAccess permissions policy.

AmazonElasticFilesSystem has been entered as a search under Permissions Policies and the AmazonElasticFileSystemClientFullAccess policy selected from the results.

Choose Next, enter AWSTransferFileSystemAdminRole for the Name and choose Create role.

Now create a role to provide Transfer Family SFTP users with the permissions required to read and write files to Amazon EFS via Transfer Family.

Follow the same steps as above for the administrative user but search for and select the AmazonElasticFileSystemClientReadWriteAccess policy when adding permissions.

AmazonElasticFilesSystem has been entered as a search under Permissions Policies and the AmazonElasticFileSystemClientReadWriteAccess policy selected from the results.

Choose Next, enter AWSTransferExternalUserRole for the Name and choose Create role.

Next, create two new IAM roles to assign to the EC2 instances hosting applications that will consume the file system.

Navigate to the IAM section of the console, select Roles, and select the Create Role button. Select AWS service as the Trusted entity type and EC2 as the Use case.

The Select trusted entity section displays a Trusted entity type set to AWS service and the EC2 check box selected under Common use cases

On the following screen, add the permissions policy for AmazonSSMManagedInstanceCore.

AmazonSSMManagedInstanceCore has been entered as a search under Permissions Policies and the AmazonSSMManagedInstanceCore policy selected from the results.

Choose Next and name the role as ApplicationOneRole before saving.

Repeat the process for a second role and name it ApplicationTwoRole.

These roles allow EC2 instances to interact with AWS Systems Manager, which is used to log in to EC2 instances in subsequent steps. However, neither role explicitly allows any actions against Amazon EFS. To enable access to the file system, we will later add statements for each role to the file system IAM policy to provide permissions against the access points.

Step 3: Creating an AWS Transfer Family SFTP server and Amazon EFS directories

Now that you have an EFS file system, use AWS Transfer Family to create an SFTP server, choosing Amazon EFS as the AWS Storage Service to store and access your data.

Once the status for your new server changes to Online, select the checkbox next to the SFTP server, and choose Add user.

Enter a Username of AdminUser, enter 0 for both the User ID and Group ID, and choose the AWSTransferFileSystemAdminRole role created earlier from the Role dropdown.

Select the EFS filesystem created earlier from the Home directory dropdown and leave the Enter optional folder field blank and the Restricted check box empty.

Under User configuration the Username of AdminUser is entered, a User ID of 0, a Group ID of 0, a Role of AWSTransferFileSystemAdminRole and Home directory of MyFileSystem.

Then add the public SSH key for the user and click Add.

In the User configuration settings the admin user’s public SSH key is entered in the SSH public keys, SSH public key field.

This admin user can now be used to recreate the on-premises directory structure in Amazon EFS.

Use an SFTP client to connect to the Transfer Family SFTP server as the AdminUser.

$ sftp -i /private/key/location AdminUser@s-abcdef01234567890.server.transfer.region.amazonaws.com
Connected to s-abcdef012345678900.server.transfer.region.amazonaws.com.
sftp> pwd
Remote working directory: /fs-0123456789abcdef0

Create a directory for a user from the first external party with POSIX user and group IDs of 2001.

sftp> mkdir ExternalOneDirectory

sftp> chown 2001 ExternalOneDirectory
Changing owner on /fs-0123456789abcdef0/ExternalOneDirectory

sftp> chgrp 2001 ExternalOneDirectory
Changing group on /fs-0123456789abcdef0/ExternalOneDirectory

sftp> chmod 0770 ExternalOneDirectory
Changing mode on /fs-0123456789abcdef0/ExternalOneDirectory

Create a directory for a user from the second external party with POSIX user and group IDs of 3001.

sftp> mkdir ExternalTwoDirectory

sftp> chown 3001 ExternalTwoDirectory
Changing owner on /fs-0123456789abcdef0/ExternalTwoDirectory

sftp> chgrp 3001 ExternalTwoDirectory
Changing group on /fs-0123456789abcdef0/ExternalTwoDirectory

sftp> chmod 0770 ExternalTwoDirectory
Changing mode on /fs-0123456789abcdef0/ExternalTwoDirectory

Confirm that the directories have been created successfully and the permissions set correctly.

sftp> ls -la
drwxrwx---   2       2001     2001     6144 Mon  D HH:MM ExternalOneDirectory
drwxrwx---   2       3001     3001     6144 Mon  D HH:MM ExternalTwoDirectory

Step 4: Adding SFTP users to AWS Transfer Family

Navigate to AWS Transfer Family, Servers in the console, select the SFTP server previously created, and choose Add user to add a user for each of our two external parties.

In the User Configuration section, create a user to represent the first external party named ExternalOneUser, and enter 2001 for both the User ID and Group ID to match the POSIX values used earlier to create the ExternalOneDirectory directory on the EFS file system.

In the User configuration settings the Username field is populated with ExternalOneUser, the User ID field with 2001, and the Group ID with 2001.

Select the AWSTransferExternalUserRole from the Role dropdown when creating each Transfer Family user.

In the User configuration settings the Role dropdown is populated with the AWSTransferExternalUserRole.

Under Home Directory, select the file system, enter ExternalOneDirectory as the home directory, and check the Restricted box. This performs what is known as a chroot operation, and in this mode, users are not able to navigate to a directory outside of the home or root directory that you’ve configured for them.

In the User configuration settings the Home directory is populated with MyFileSystem, the optional directory dropdown is populated with ExternalOneDirectory and the Restricted check box is selected.

If your directory structure requirements are more complex than a simple home directory, you can use logical directories to provide a virtual directory structure with user-friendly names. The use of logical directories is described in the blog post Simplify your AWS SFTP Structure with chroot and logical directories.

Finally, add the public SSH key for the external user and click Add.

In the User configuration settings the external user’s public SSH key is entered in the SSH public keys, SSH public key field.

Repeat the process to create a new user named ExternalTwoUser, but use the User ID of 3001 and Group ID of 3001 to match the POSIX configuration for the ExternalTwoDirectory created previously and set the Home directory value to ExternalTwoDirectory.

Now when you connect over SFTP as ExternalOneUser, you are able to upload a dummy file.

$ touch VendorOne.data
$ sftp -i /private/key/location ExternalOneUser@s-abcdef01234567890.server.transfer.region.amazonaws.com
Connected to s-abcdef012345678900.server.transfer.region.amazonaws.com.
sftp> put VendorOne.data
Uploading VendorOne.data to /VendorOne.data

However you are constrained to the ExternalOneDirectory as your root directory.

sftp> pwd
Remote working directory: /
sftp> ls
VendorOne.data
sftp> cd ..
sftp> ls
VendorOne.data

When you connect as ExternalTwoUser, you can only see the contents of ExternalTwoDirectory, which is set as the root directory for that SFTP user.

$ sftp -i /private/key/location ExternalTwoUser@s-abcdef01234567890.server.transfer.region.amazonaws.com
Connected to s-abcdef01234567890.server.transfer.region.amazonaws.com.
sftp> pwd
Remote working directory: /
sftp> ls
sftp>

This demonstrates that SFTP users are contained within the appropriate part of the EFS file system.

Step 5: Creating EFS Access Points to enable access for EC2 hosts

Create an access point for each of two applications, one that should only have access to data from the first external party and another that will be able to access the data uploaded by both external parties.

Choose Create Access Point in the Elastic File System, Access Points section of the EFS console.

The Create access point button is located in the top right corner of the Elastic File System, Access points section of the console.

Enter details of the Amazon EFS file system in the resulting screen, set the Name for the access point as ApplicationOneAccessPoint, and the Root directory path that the application should be constrained to as ExternalOneDirectory.

Under Details the file system created earlier has been selected from the File system field dropdown, the Name field is set to ApplicationOneAccessPoint and the Root directory path field is set to /ExternalOneDirectory.

In the POSIX user section, add 4001 as the User ID and Group ID, and in Secondary group IDs, enter 2001, the POSIX group ID associated with the ExternalOneDirectory directory previously created on the Amazon EFS file system.

Under POSIX user settings the User ID is set to 4001, the Group ID is set to 4001 and the Secondary group IDs field is set to 2001.

Leave the remaining fields blank and create the access point.

Repeat the process to create an access point for the second application, but assign a Root directory path of /, User ID of 5001, Group ID of 5001, and Secondary group IDs of 2001,3001.

Under Details the file system created earlier has been selected from the File system field dropdown, the Name field is set to ApplicationTwo AccessPoint and the Root directory path field is set to /. Under POSIX user settings the User ID is set to 5001, the Group ID is set to 5001 and the Secondary group IDs field is set to 2001,3001.

Step 6: Updating the Amazon EFS file system IAM policy to enable access points

Update the EFS file system IAM policy by adding extra statements to only allow certain actions against our file system when accessed via the access points and to restrict the use of each access point to only the IAM roles created for each application.

Edit the following JSON to replace the ARNs for each role and access point to match those you have created in your environment.

,
{
    "Sid": "ApplicationOneAccess",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::111122223333:role/ApplicationOneRole" },
    "Action": [
        "elasticfilesystem:ClientMount",
        "elasticfilesystem:ClientWrite"
    ],
    "Condition": {
        "StringEquals": {
            "elasticfilesystem:AccessPointArn": "arn:aws:elasticfilesystem:region:111122223333:access-point/fsap-1234567890abcdef0"
        }
    }
},
{
    "Sid": "ApplicationTwoAccess",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::111122223333:role/ApplicationTwoRole" },
    "Action": "elasticfilesystem:ClientMount",
    "Condition": {
        "StringEquals": {
            "elasticfilesystem:AccessPointArn": "arn:aws:elasticfilesystem:region:111122223333:access-point/fsap-2345678901abcdef0"
        }
    }
}

These statements allow EC2 instances with the ApplicationOneRole assigned to mount and write to the file system via the first access point, but those with ApplicationTwoRole to only mount the file system when using the second access point.

Navigate to the Amazon EFS console, select File Systems, then select the previously created file system. Choose the File system policy tab, and select Edit.

The File system policy tab is located beneath the General settings section for the selected file system and the Edit button to the right on the Policy tab.

Add the updated JSON to the array of policy statements.

The updated JSON is inserted into the EFS filesystem IAM policy at the end of the array.

Step 7: Mounting Amazon EFS via access points from Amazon EC2 instances

Launch an Amazon EC2 instance with the previously created ApplicationOneRole role into the same VPC used to create the EFS file system and use AWS Systems Manager to connect to the instance.

Install the amazon-efs-utils package and create a directory that you will use as the EFS file system mount point using the following commands:

AppOneHost$ sudo yum install -y amazon-efs-utils
AppOneHost$ sudo mkdir /mnt/efs

Note that the instance is not able to mount the EFS file system directly:

AppOneHost$ sudo mount -t efs -o tls,iam fs-0123456789abcdef0:/ /mnt/efs
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'

Or access the file system via the access point requiring the ApplicationTwoRole role.

AppOneHost$ sudo mount -t efs -o tls,iam,accesspoint=fsap-2345678901abcdef0 fs-0123456789abcdef0:/ /mnt/efs
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'

An EC2 instance with the ApplicationOneRole assigned is only able to mount the file system through the access point created for that role. When using that access point, the ExternalOneDirectory directory is presented as the root directory.

An application running on this instance can both read from and write files to the directory, because the file system IAM policy allows both the mount and write actions for the role when using the access point. In addition, the access point contains a secondary group ID with matching file system permissions.

AppOneHost$ sudo mount -t efs -o tls,iam,accesspoint=fsap-1234567890abcdef0 fs-0123456789abcdef0:/ /mnt/efs
AppOneHost$ touch /mnt/efs/AppOne.data
AppOneHost$ ls -la /mnt/efs/
total 8
drwxrwx--- 2 2001 2001 6144 MON  D HH:MM .
drwxr-xr-x 4 root root   29 MON  D HH:MM ..
-rw-r--r-- 1 2001 2001    0 MON  D HH:MM VendorOne.data
-rw-r--r-- 1 4001 4001    0 MON  D HH:MM AppOne.data

Now launch a second EC2 instance with the ApplicationTwoRole assigned and observe that only the second access point can be mounted and both ExternalOneDirectory and ExternalTwoDirectory are available because the root directory path was set to the file system root when creating the access point.

AppTwoHost$ sudo mkdir /mnt/efs
AppTwoHost$ sudo mount -t efs -o tls,accesspoint=fsap-2345678901abcdef0 fs-0123456789abcdef0:/ /mnt/efs
AppTwoHost$ ls /mnt/efs
ExternalOneDirectory  ExternalTwoDirectory

This second host can read the contents of both directories because the file system policy statement for the access point contains mount permissions, and the file system group assigned read rights for each directory is contained within the secondary group IDs for the access point.

However, the host is unable to write to either directory even though the access point has secondary group IDs that match those with write permissions to those directories because the file system IAM policy only allows the EFS ClientMount action for the ApplicationTwoRole IAM role against the access point and not the ClientWrite action.

AppTwoHost$ sudo touch /mnt/efs/ExternalOneDirectory/AppTwo.data
touch: cannot touch ‘/mnt/efs/ExternalOneDirectory/AppTwo.data’: Read-only file system

AppTwoHost$ sudo touch /mnt/efs/ExternalTwoDirectory/AppTwo.data
touch: cannot touch ‘/mnt/efs/ExternalTwoDirectory/AppTwo.data’: Read-only file system

These examples have shown how you can replace NFSv4 ACLs used on-premises and restrict access to data transferred by SFTP using combinations of:

  • AWS Transfer Family POSIX profiles and home directories.
  • Amazon EFS file system permissions and IAM policies.
  • Amazon EFS access point POSIX profiles and root directories.

Cleaning up

If you’ve been following along with the steps in this post, then don’t forget to delete the resources set up to avoid any future recurring charges, including your EC2 hosts, AWS Transfer Family SFTP server, Amazon EFS file system, and IAM roles.

Conclusion

In this post, we showed you how Tata Consultancy Services was able to build a secure file transfer solution for the Foxtel Group using AWS Transfer Family and Amazon EFS.

We were able to lock down access to sensitive datasets using a combination of AWS Transfer Family users with home directories and POSIX profiles alongside Amazon EFS access points with POSIX profiles and a file system IAM policy. This approach replaced the NFSv4 ACLs that were present in the on-premises platform and enabled a seamless migration without any compromise in security.

Thanks for reading this blog. Leave a comment in the comments section if you have any questions. To learn more about the components of this solution, please check out the following links:

Sean Byrne

Sean Byrne

Sean is an Enterprise Solutions Architect at Amazon Web Services who works with Media and Entertainment companies. When he is not working, he can usually be found in the surf or at the beach with his two children.

Mohammed Usman

Mohammed Usman

Usman is an AWS Certified Solution Architect and Infrastructure Services Consultant with TCS based in Sydney. He works with customers to provide cloud migration solutions and helps them to accelerate their journey to the cloud.