Customize file delivery notifications using AWS Transfer Family managed workflows

Enterprises and organizations that transfer files, both internally and with external parties, can benefit from having file-processing protocols that facilitate efficient and transparent file-transfer management. When transferring files, especially at scale, it is important to have visibility into the status of file transfer, and for you to able to process files as you receive them. There are also situations where you need visibility and actionability on files that did not transfer completely, or that only partially uploaded, so that you can take troubleshooting steps or receive proper notification. Without such visibility, files you need can get lost in the shuffle after failing to fully transfer, and workloads can stall or suffer.

AWS Transfer Family is a secure transfer service that enables you to transfer files into and out of AWS storage services. Customers using AWS Transfer Family to transfer files often perform preprocessing and validation actions before the files are ingested by downstream applications. This includes managed workflows that look for malware, validate file and content formatting, manage decryption, or meet security and compliance requirements.

In this blog, we will demonstrate how AWS Transfer Family supports managed workflows for file processing, which enables you to create, automate, and monitor your file transfers over SFTP for both complete and partial file uploads. This feature enables you to coordinate all the necessary steps required for file processing, such as copying, tagging, and deleting. Prior to today, this was only possible for complete file uploads. Now it is possible to define separate workflows for disrupted file uploads. When uploading to a server, especially with the validation steps that may pass or fail, the template provided in this blog demonstrates how customers that are uploading the files via AWS Transfer Family can send notifications acknowledging the file transfer is successful or otherwise notifying the user if the file upload has failed.

Solution overview

Files that are transferred over SFTP and made available via Amazon S3 and Amazon EFS can then be stored, processed, or archived for long-term storage. This solution shows you how you can use AWS Transfer Family workflows with a custom AWS Lambda function to send notifications on a file upload.

You can configure an AWS Transfer Family workflow to use a custom Lambda to notify when a file has been successfully transferred. Predefined steps such as copy, tag, and delete can be used to orchestrate a file copy from one S3 bucket to another. It also offers the ability to tag and delete the original transferred file. If any of the steps fail, the workflow will branch to the exception workflow to handle this.

The following diagram summarizes the workflow of the solution:

Figure 1: Architecture overview describing the workflow to transfer a file into Amazon S3 including the predefined steps to copy, tag and delete objects

1. A Transfer Family user initiates a file transfer to the Transfer Family server via SFTP. In this example, we will refer to the user as user1.

2. The file is transferred into Amazon S3 in our upload bucket under the user’s path. In our example, the path is /user1/.

3. Transfer Family orchestrates the post-upload workflow execution. The workflow comprises the following steps:

3.1.  Transfer Family workflow invokes a Lambda function that queries an Amazon DynamoDB table. The DynamoDB table contains usernames and their notification preference of email, HTTPS, or SMS.
3.2.  If the user’s preference does not exist in the DynamoDB table, it creates an Amazon Simple Notification Service (Amazon SNS) subscription for the user.
3.3.  The AWS Lambda function publishes to Amazon SNS to notify the user based on their preferred endpoint defined in the Amazon DynamoDB table. In this example, the user’s preference is to be notified via email.
3.4.  Copy step: The S3 object in the user’s home directory is copied from the S3 bucket for upload to the S3 bucket for application. In this example, the object from S3 upload /user1/ is copied to Amazon S3 application /user1/ home directory.
3.5.  Tag step: The Amazon S3 object tags “Notification: Successful” and “Copied: Successful” are applied to the object under the path /user1/ in S3 application bucket to indicate the notification has been sent to user.
3.6.   Delete step: The object is deleted from the S3 bucket for application. In our example, it is deleted from the /user1/ home directory.

4. Exception step: Any exceptions in the nominal steps in the shared workflow will be handled using the exception workflow. The exception workflow comprises a step to copy the file to the quarantine bucket. The file will then be tagged, and the original file will be deleted from the upload bucket.

Some customers may want more granular control on processing of uploaded files by defining separate processing steps for files that are fully uploaded compared with files that are only partially received. For example, a customer may not want to send a notification on every successful upload but may want to send a notification in the event an incomplete upload occurs.

Previously, customers could only trigger Transfer Family workflows for fully uploaded files to ensure data integrity for downstream data lakes and applications. However, to provide more control to customers on how to handle incomplete file uploads, Transfer Family recently also launched the ability to specify and execute a separate workflow for partially uploaded files. Using this feature, you can modify the architecture in this blog post to only send a notification in the event when a file is partially received, for instance, due to premature session disconnect.

Figure 2: Create a managed workflows for complete and partial uploads

Getting started

This section provides you with the high-level steps to get started with this solution.

Prerequisites

We assume you have an SFTP server setup and a user created. Follow steps 1–3 from the Getting started with AWS Transfer Family tutorial to proceed.

1. Deploy AWS CloudFormation template

AWS CloudFormation automates the setup of the solution for you. The following components are provisioned by this CloudFormation template:

• S3 buckets to store the files
• Transfer Family–managed workflow to orchestrate the file processing
• A custom Lambda to send notifications as part of the managed workflow
• Amazon SNS with the endpoints to send the notifications to
• DynamoDB to store user preferences for notifications
• Amazon CloudWatch for monitoring
• AWS Identity and Access Management (IAM) roles

Figure 3: Create a CloudFormation stack to deploy the template

To deploy the template:

1. Open the AWS CloudFormation console.
2. Launch a new stack by downloading the CloudFormation template. To create the stack, follow the steps in Creating a stack on the AWS CloudFormation console. Alternatively, click on this stack to launch it in Oregon Region using the CloudFormation console.
3. provide a name for the stack. In this example, we have named the stack TransferFamily-blog-stack.

When testing the stack in Oregon Region, the CloudFormation template took approximately 2 minutes to deploy.

Currently, the template file provided can be deployed directly to Oregon Region. To deploy to other Regions:

1. Download the CloudFormation template, notification Lambda .zip file, and DynamoDB Lambda .zip file.
2. Open the CloudFormation template and edit the name of the S3 bucket on lines 127 and 311, as shown here:

Code:
 S3Bucket:
  Fn::Sub: awsstorageblogresources

3. Make sure that you have the correct S3Key on lines 128 and 312 as per where you are planning to upload these files.
4. Select your preferred Region from the navigation bar on the AWS Management console.
5. Create or select an S3 bucket that you can upload the templates to.
6. Upload the three files to your S3 bucket.
7. Deploy the stack as per the steps above using the CloudFormation console.

The Outputs tab inside the created stack details will contain the values that you will need in the next steps for the managed workflow ID, IAM roles, S3 buckets, and DynamoDB table. Once you have deployed your CloudFormation stack, the next step is to assign your workflow to your AWS Transfer Family server.

2. Add workflow to your Transfer Family server

The following steps are to modify your AWS Transfer SFTP server and to attach the ready workflow to it:

Figure 4: Modify the SFTP Server created by the CloudFormation stack

1. Open the AWS Transfer Family console.
2. Select the Server that you created as part of the prerequisite step.
3. In the Additional details section, select edit.
4. For Managed workflows, attach the workflow with description as Workflow for blogpost and IAM role starting with TransferFamily-blog-stack-ManagedWorkflowRole{uid} that was created by the deployed CloudFormation stack as shown here:

Figure 5: Attach the managed workflow created by the CloudFormation stack to your server

Next, we need to add users to the managed workflow that was created by the CloudFormation stack.

Figure 6: Add service-managed users to the SFTP that you created

3. Add Amazon S3 service-managed users to your SFTP server

To add a service-managed user to the SFTP server that you created:

1. On the AWS Transfer Family page, select Servers from the menu.
2. Select the check box of the server that you created in the prerequisite step.
3. Choose Add User.
4. In the User Configuration section, enter the username user1.

Figure 7: Add user configuration to the SFTP server

5. For Access, choose the IAM role starting with TransferFamily-blog-stack-RoleforTransferFamilyUser-{uid}.
6. For Home directory, choose the Amazon S3 bucket starting with upload-. The bucket name is provided under the UploadBucketName key in the CloudFormation stack output) and fill user1 as the This is where user1 will land when they sign in using the client application.
7. For Restricted, select the check box so that user1 can’t access anything outside of that folder and can’t view the S3 bucket or folder name.
8. For SSH public key, enter the public SSH key part of the SSH key pair. To find more information on how to generate an SSH key pair, see Generate SSH keys.

Repeat the steps to add users. For example, a second user with username as user2.

The final step of the setup is to update the user preferences in the DynamoDB table. The CloudFormation stack prepopulates the user preferences with sample data.

4. Setup user notification preference in Amazon DynamoDB table

The DynamoDB table contains user preferences, including the username, endpoint, and protocol. This information will be used by the custom Lambda function to send notification based on user preference. There are three supported endpoint values that can be tested: email, SMS, and HTTPS.

To update the DynamoDB table with your user preference:

Figure 8: DynamoDB table showing the configured user preference

1. Open the Amazon DynamoDB console.
2. Choose Tables.
3. Select the table starting with TransferFamily-blog-stack-NotifyUserPreference{uid}.
4. Choose Explore Table Items.
5. Under Items Returned, select user1.
6. Select Actions and choose Edit item from the dropdown.
7. Update the endpoint value to your own email address.
8. Choose save changes.

(Optional) Note that to test SMS, there are additional steps and setup costs. To get started, update the preferred phone number in the DynamoDB table. For more information on how to set up and send SMS messages, follow the steps to Adding and verifying phone numbers in the SMS sandbox

It is important to note that the user value should match the same usernames added to the SFTP server in the previous step.

5. Transfer a file using a client

To start testing the file transfer via SFTP, follow step 4 in the Getting started with AWS Transfer Family tutorial.

When a user uploads a file for the first time, a new subscription will be added to the Amazon SNS topic. You must confirm the subscription before the email address can start receiving messages. This is handled by the managed workflow and the custom Lambda function. The uploaded file will then appear in the application bucket.

Note: you will encounter a failure when you first transfer a file unless you can manually add a subscription to the Amazon SNS topic using the AWS Management Console. Additionally, there could be a delay in receiving the confirmation request once the initial file has been uploaded.

Logging and monitoring

Here are some options for keeping an eye on your transfer workloads.

AWS CloudTrail

You can monitor your Transfer Family SFTP API calls using AWS CloudTrail. This allows you to gain insight into API activity which can help you meet your security and compliance requirements. All Transfer Family actions are logged by CloudTrail.

For example, calls to the CreateServer, ListUsers, and StopServer actions generate entries in the CloudTrail log files. You can view this in the AWS CloudTrail console.

Amazon CloudWatch

You can use Amazon CloudWatch logging to monitor, store, and access your log files. To enable logging, you start by creating an IAM policy that enables CloudWatch logging. You then create an IAM role with a Transfer Family trust policy and attach the policy to it. CloudWatch is enabled during the SFTP server setup when the logging role AWSTransferLoggingAccess is created. For more information, refer to Log activity with CloudWatch in the AWS Transfer Family User Guide.

Figure 9: CloudWatch logs showing the copy step completion

Amazon CloudWatch log groups can be used for monitoring and troubleshooting your environment. CloudWatch Logs receives log events from Transfer Family and Lambda. To create a CloudWatch dashboard for monitoring your Transfer Family environment, refer to the Monitoring your AWS SFTP environment blog post.

Cleaning up

If you have finished using the resources that have been created as part of this blog post, don’t forget to delete them in order to avoid incurring future charges.

Amazon S3

1. You can use the Amazon S3 console to empty the bucket, which deletes all of the objects.
2. Select the bucket name starting with upload-’, and choose Empty.
3. On the Empty bucket page, confirm that you want to empty the bucket by entering the bucket name and choosing Empty.
4. Repeat the process for bucket names, starting with application- and quarantine-.

AWS Transfer Family

To disassociate the workflow from the server, open the AWS Transfer Family console.

1. Choose Servers.
2. Select the server that was created.
3. Under Managed workflows, change the Workflow from the dropdown to “–“.
4. In the confirmation box, enter delete to confirm that you want to delete the server.

To delete users associated with a workflow

1. Choose Servers.
2. Select the server that was created.

1. Select the user that you choose the delete.
2. Select Actions and choose Delete from the dropdown.

AWS CloudFormation

Figure 10: CloudFormation console showing the delete stack option

1. The next step is to delete the stack. Open the AWS CloudFormation console.
2. Select the stack that was created and choose Delete.
3. Select Delete Stack when prompted.
4. The CloudFormation stack takes approximately 2 minutes to delete.

Conclusion

In this blog post, we covered how Transfer Family managed workflows can be used to send acknowledgments to a user upon a complete file upload. We did this with a workflow that included a custom step with a Lambda function that checked user notification preferences and sent a message to the users via email or SMS. This post demonstrates predefined steps, such as copying a file from Amazon S3 using the username as the path name, tagging the processed file, and deleting the original file as a part of your post-upload file processing actions.

Customers that used Transfer Family to transfer files want a mechanism to define custom steps such as sending notifications on a successful file upload after performing preprocessing validation actions. The user uploading the file will then get a notification to know if the upload was successful and then can retry and fix any issues if it was unsuccessful. That will eventually increase the number of successful uploads.

We invite you to try the managed workflows feature of AWS Transfer Family and to deploy and try the provided template.

Thanks for reading the blog post!