Improve throughput for internet facing file transfers using AWS Global Accelerator and AWS Transfer Family services

Remote work has become more common over the last couple of years. For IT teams, this accelerated the need to support the secure distribution and collection of files and software updates for global teams in multiple locations. Additionally, organizations continue to partner with vendors or 3rd parties to distribute or collect data for analytics or reporting needs. For either group, employees or external partners, using the right solution to reduce latency is key when transferring files. The AWS Transfer Family offers Secure Shell File Transfer Protocol (SFTP) and File Transfer Protocol over SSL (FTPS) endpoints to support file transfers between the AWS cloud, vendors, and a globally dispersed workforce

When users are distributed globally, a common challenge these SFTP/FTPS workloads face is longer than acceptable latency for uploads and downloads. This can get especially aggravating when transferring large files, such as software updates.

As explained in this previous post about how to minimize network latency with your AWS Transfer for SFTP servers, customers have the option to replicate data and host it in multiple AWS Regions to improve performance. There are circumstances, especially related to disaster recovery (DR), ultra-high availability, and regulations, where geographically-isolated application replication is a necessary strategy. In other cases, a cost-effective and simple to manage alternative is preferred. This blog covers a solution for the latter scenarios.

This solution will use AWS Global Accelerator and AWS Transfer Family to improve data transfer throughput resulting in faster transfer times. Please refer to our test results at the tail end of the blog.

The AWS Transfer Family provides fully managed support for file transfers directly into and out of Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS). It supports Secure File Transfer Protocol (SFTP), File Transfer Protocol over SSL (FTPS), and File Transfer Protocol (FTP). Getting started with the AWS Transfer Family is easy; you no longer need to deploy or manage your own physical infrastructure.

AWS Global Accelerator is a networking service that sends your user’s traffic through Amazon Web Service’s global network infrastructure, improving your end-user performance by up to 60%. When the internet is congested, Global Accelerator’s automatic routing optimizations will help keep your packet loss, jitter, and latency consistently low.

With Global Accelerator, you are provided with two global static IPs to simplify traffic management. On the back end, you can add or remove your AWS application origins, such as Network Load Balancers, Application Load Balancers, Elastic IPs, and EC2 Instances, without making user-facing changes. To mitigate endpoint failure, Global Accelerator automatically re-routes your traffic to your nearest healthy available endpoint.

In this post, we walk through deploying an AWS Global Accelerator in front of an AWS Transfer Family endpoint located in us-east-1 Region. We then test and measure the throughput with and without AWS Global Accelerator. For our example, we’ll use SFTP endpoints and access these endpoints from (Mumbai). All three locations are outside of the AWS network so that we can test improvements in overall throughput.

For this solution, we will start by creating an SFTP endpoint which is backed by an Amazon S3 bucket and has a public facing Elastic IP (EIP) endpoint. AWS Global Accelerator requires a public facing EIP The SFTP endpoint we will be creating is a VPC endpoint with internet-facing access.

Prerequisites

For this walkthrough, you should have a working knowledge of Amazon VPCs and Elastic IP Addresses (EIP). You should also be able to create Amazon S3 Buckets and SFTP Servers using AWS Transfer Family.

Overview of solution

The diagram below showcases the overall solution architecture.

The steps below will walk you through creating the solution as outlined in the architecture diagram above.

Step 1:  Create an Amazon S3 Bucket : Provide an Amazon S3 bucket when setting up your file transfer protocol-enabled server. AWS Transfer Family will use your Amazon S3 bucket to service your users’ transfer requests. You can use an existing bucket, or you can create a new one. For information on creating a new bucket, see How do I create an S3 bucket? in the Amazon Simple Storage Service Console User Guide.

Step 2: Follow the link to Allocate an Elastic IP Address in a Region where you want to deploy your SFTP endpoint.

Step 3: Follow the link to Create an internet-facing SFTP-enabled server; details and how-to steps linked.

• For this exercise, make sure that you choose VPC Hosted endpoint option on “Choose an endpoint” screen.
• For the identity provider, this walkthrough has chosen Service Managed, but you can also choose Custom.
• For customer hostname, choose None.
• Next when you choose the VPC and subnets, make sure you use the EIP you created in previous step during the IPv4 address selection.
• Make sure that the subnet has access to the internet and the security group (SG) attached to it allows traffic on port 22 from internet. shots are shown below:

• After you have created the server, the following image with Endpoint configuration will show up. To see them, click on the server ID in console, the server status shows “Online”. This shows the private and public IPv4 address of the endpoint. Make a note of the public IPv4 address, we shall use it later to test.

• Optionally, you can apply fine-grained permissions to each SFTP user.

Step 4: Create a standard Accelerator with AWS Global Accelerator.  By using a standard accelerator, you can improve performance of your internet applications that are used by a global audience. With a standard accelerator, Global Accelerator directs to the nearest AWS global network edge location to Region of the data residency.

• As shown in below image, enter the name of the accelerator You can also bring your own IP Address

• Next, as shown in the following image, on the Add page input the following values.
  • Listener port : 22
  • Protocol : TCP
  • Client affinity : None

• An accelerator includes one or more listeners that direct traffic to one or more endpoint groups. An endpoint group includes endpoints, which in our case is the EIP for the SFTP server endpoint we created earlier. Choose the Region where our SFTP endpoint was created, which is us-east-1 in this case. Here all traffic is routed to one endpoint in one Region. You can also choose to load balance traffic across multiple SFTP endpoints for load balancing; make sure to adjust traffic dials and weights accordingly.

• For the endpoint configuration, choose the endpoint type as Elastic IP Address and then pick the EIP which was allocated to the SFTP endpoint created earlier. Click on create accelerator.

Once the accelerator status is deployed, make sure that the status for all the listeners are healthy before you proceed to testing the performance of your SFTP downloads from various locations across the globe. A screenshot below shows the final status.

Performance Metrics

With this workflow setup, we tested the performance improvements.

For our testing we are looking at the following factors.

Bucket Location: us-east-1
SFTP Endpoint: us-east-1
File size: 100 MB

We performed the tests with client’s location in London, Mumbai, and Sydney. Note that these client endpoints are outside of the AWS network and use the local home/last mile network. For testing purpose, a public VPN service is used to simulate the client location. The result might be different depending on the internet bandwidth, SFTP client configuration and actual physical location of the client. We set this test up to compare the performance of directly connecting to an SFTP endpoint with connections using AWS Global Accelerator.

Following tables summarizes, the results.

Note that the source IP of your clients are not preserved or be recorded to your AWS Transfer Family CloudWatch Logs. To view source IP of your end users, you will need to use VPC Flow Logs.

Cleaning up

To avoid incurring future charges, delete the AWS Global Accelerator standard accelerator.

Next delete the SFTP server, EIP and the S3 bucket created in the solution steps defined earlier.

Conclusion

Whether your users are in remote locations or you need to source data from devices across the globe into AWS over SFTP, this post shows you how you can provide improved throughput for such internet facing SFTP workloads using AWS Transfer Family and AWS Global Accelerator. We created an internet facing SFTP server endpoint using the VPC mode and assigned it an EIP which is one of the endpoints types supported by AWS Global Accelerator. We created an AWS Global Accelerator and integrated it with the SFTP endpoint.

After building this solution, we tested download and upload speeds with and without using AWS Global Accelerator. On average, we found that downloads were improved by 120%, while uploads were improved by 45%.

About Authors