­­Encrypt and decrypt files with PGP and AWS Transfer Family

1/11/2024: Updates made due to CloudShell migration to Amazon Linux 2023 (AL2023).

Protecting sensitive data is not a novel idea. Customers in industries like financial services and healthcare regularly exchange files containing sensitive data, including Personal Identifiable Information (PII) and financial records with their users. Pretty Good Privacy (PGP) encryption of these files is often a key requirement for regulatory and data policy compliance.

PGP is a popular encryption system that provides an additional layer of protection for sending emails and files. It makes sure that only intended parties have access to your data in these files. In the past, one of the main challenges that customers faced while using PGP-based encryption in the AWS Cloud was the added layer of operational complexity associated with decrypting files.

In a previous post, we explain how to enable password authentication for AWS Transfer Family using AWS Secrets Manager. In this post, we build on top of that architecture. We generate a PGP key pair, store the private key and other custom variables in Secrets Manager, and create a Transfer Family managed workflow. Then, we explore how to encrypt, automatically decrypt, and handle exceptions within Transfer Family managed workflows.

Transfer Family is a fully managed service that makes it easy to deploy file transfer workloads on AWS. In December 2022, Transfer Family announced built-in support for PGP decryption of files uploaded over SFTP, FTPS or FTP protocols to Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS). Now, customers can receive PGP encrypted files from their users via Transfer Family and automatically decrypt them using a predefined decrypt workflow step. Let’s get started!

Solution Overview

For this solution, we create an AWS Transfer Family managed workflow that will decrypt and store incoming files to Amazon S3. These files will then be automatically copied to another prefix called “archive” and the original files will be deleted from the user’s home directory. If any errors occur during upload, a workflow exception handler will be triggered, the file quarantined, and a email alert sent out. The blog will be broken down into the following sections.

1. Generate & export a PGP key pair
2. Store Transfer Family user configurations in Secrets Manager
3. Create a managed workflow with an exception handler
4. Associate the workflow with the Transfer Family server
5. Create an Admin alert
6. Test the workflow

Prerequisites

For this post, we provide an AWS CloudFormation template that deploys the following resources. Note that deploying the CloudFormation stack is optional. Download the template here.

Resources deployed by the CloudFormation template

• Transfer Family server
• Amazon S3 bucket
• Amazon Simple Notification Service (Amazon SNS) topic
• Two AWS Lambda Functions (for authentication and exception handling)
• Five AWS Identity and Access Management (IAM) roles (for Amazon CloudWatch logging, Lambda Execution, SFTP access to S3, and for the Transfer Family managed workflow)

PGP key pair

This is a two-step process. First, we generate a PGP key pair, and then we export the newly generated key pair into a file.

Generate the PGP key pair

The process for generating a PGP key pair depends on your operating system. For this post, we use AWS CloudShell to create our key pair with GNU Privacy Guard (GPG), a free implementation of the OpenPGP standard. Note that AWS CloudShell is not supported in all AWS Regions. Check here to see if your Region is supported. AWS CloudShell runs on Amazon Linux 2023, for Unix, Windows, or macOS operating systems check here to see supporting commands and documentation. To generate the key pair:

1. Install GPG by running the following command:

sudo dnf install --allowerasing gnupg2-full

2. Generate a key pair by entering the following command:

gpg --full-generate-key

2.1. You are prompted to enter certain specifications for your key pair. When prompted to “Please select what kind of key you want” hit ‘1’ and then “Enter”, which selects option 1 RSA. As a reminder, RSA is a public-key encryption system that encrypts data with asymmetric encryption.

2.2. For this post, we accept the default key size of 3072 bits. When prompted with “What keysize do you want?”, hit “Enter”, which causes 3072 bits to be chosen.

2.3. Next you are asked “Key is valid for?” For this example, we hit “Enter” which means the key will never expire.

3. Now, you are prompted to construct a user ID to identify your key pair. You must provide:

3.1. “Real name”, we use testuser for this example.

3.2. “Email”, enter the email you would like to be associated with this key pair. We use this email later when we encrypt a file. For this example, we use testuser@example.com.

3.3. Verify the information you entered and accept typing “O” for Okay.

3.4. “Passphrase”, make sure you write down your passphrase so you don’t forget it. We need it later.

When your key pair has been created, gpg outputs a “pub”, “uid”, and “sub”.

Export the PGP key pair

1. Let’s export our private key to a file by running the following command:

gpg –output <file name here> –armor –export-secret-key <email here>

For this example, we use:

gpg --output testuser-gpg --armor --export-secret-key testuser@example.com

2. Run the following command to view your PGP private key:

cat testuser-gpg

3. Copy the key from -----BEGIN PGP PRIVATE KEY BLOCK----- all the way to -----END PGP PRIVATE KEY BLOCK-----. We use it later.

Store your user configuration in Secrets Manager

1. In a new tab, navigate in the console to Secrets Manager. Don’t forget to make sure you are in the same Region as your Transfer Family server. We store our user configuration, including the newly exported private key, in Secrets Manager by selecting either Secrets or Store a new secret.
2. On the Choose secret type page, for Secret type, choose Other type of secret.
3. In the Key/value pairs section, choose the Key/value We add the PGPPrivateKey, PGPPassphrase, Password, HomeDirectoryDetails, and the arn of the IAM Role created by CloudFormation allowing SFTP access to Amazon S3.

Key: Enter PGPPrivateKey.

Value: Paste the text of your private key that you previously copied.

Select +Add row.

Key: Enter PGPPassphrase.

Value: Enter the passphrase you used when you generated your PGP key pair.

Select +Add Row.

Key: Enter Password.

Value: Create a password for your user.

Choose +Add Row.

Key: Enter HomeDirectoryDetails.

Value: Enter [{“Entry”: “/”, “Target”: “/<name of s3 bucket created by CloudFormation>/<yourusername>”}]. You can find the bucket name in the Output section of the CloudFormation Stack.

Choose +Add Row again.

Key: Enter Role.

Value: Enter the ARN of the IAM role called SftpS3AccessXXXXXXXXX created by the CloudFormation Template. You can find the IAM Role’s ARN in the Output section of the CloudFormation Stack.

Your Secret should look similar to the following image. Leave the Encryption key section as default, and choose Next.

4. On the Configure secret page, enter a name and description for your secret.

If you’re creating a default key, which is a key that can be used by any Transfer Family user for a specific Transfer Family server, then enter aws/transfer/<server-id>/@pgp-default. Replace <server-id> with the ID of the Transfer Family server created by CloudFormation. This ID can be found in the Outputs section of your CloudFormation stack.

For this post, we create a key for a specific SFTP user, so we enter aws/transfer/<server-id>/<username>. The ID of the Transfer Family server created by CloudFormation can be found in the Outputs section of your CloudFormation stack. Replace <username> with a username of your choice.

5. Accept all default settings, and choose Next.
6. On the Configure rotation page, accept all the default settings, and choose Next.
7. On the Review page, choose Store to create and store the secret.

Your new secret should look similar to the following image when you select it. We are now done in Secrets Manager, great job!

Create a managed workflow in Transfer Family

So far, we have created a PGP key pair and stored our private key and user configuration in Secrets Manager. Now, we create a managed workflow in Transfer Family. By orchestrating file-processing tasks, managed workflows preprocess data before it’s consumed by your downstream applications. In our workflow, we decrypt incoming files and store them in an S3 bucket. We also create an exception handler to deal with situations where files arrive already decrypted or throw an error during processing. Let’s begin!

Create a workflow

1. In the AWS Management Console, use the search bar to navigate to Transfer Family.
2. In the left navigation pane, choose Workflows, then choose Create workflow.
3. On the Create workflow page, enter a description. This description appears on your finished Workflows.

Managed workflows have two types of steps: Nominal steps and Exception-handling steps. Nominal steps are file-processing steps that you want to apply to incoming files. Each Nominal step you add is processed in a linear sequence. Exception-handling steps execute if any of the nominal steps fail. Let’s create a nominal step that decrypts our incoming files and archive the original unencrypted file in a different prefix in our S3 bucket.

Configure Nominal steps

1. Create the “decrypt” step.

1.1. In the Nominal steps section, choose Add step.

1.2. Select Decrypt file, then choose Next.

1.3. Under Configure parameters, choose a Step name. Make sure there are no spaces in your name. For this example, we use "decrypt".

1.4. Choose Amazon S3 as Destination for decrypted files, then select the S3 bucket created by the CloudFormation template from the dropdown list Destination bucket name. The bucket name is pgp-blog-s3-bucketXXXXXXX.

1.5. As Destination key prefix for our decrypted files, we use:

decrypted-files/${transfer:UserName}/${transfer:UploadDate}/

${transfer:UserName} This value parameterizes the destination prefix by username. Every time a new user uploads a document, a new prefix is created in the S3 bucket.

{transfer:UploadDate} This value parameterizes the destination prefix by the upload date. Since this value comes after the user name parameter, this is a prefix within each user. Each day a user uploads a new file, a new prefix is created with the date of the upload in the user name folder.

We leave the Overwrite as default, and choose Next.

1.6. In the Review and create page, review your selections, and choose Create step.

2. Create the “archive” step.

The second step copies the original encrypted file to the key prefix: archived-files/${transfer:UserName}/${transfer:UploadDate}/

2.1. In the Nominal steps section, choose Add step.

2.2. On the Choose step type page, select Copy file, and then Next.

2.3. Under Configure parameters, choose a Step name. For this example, we choose "archive"

2.4. As File location select Copy the original source file to a new location.

2.5. As Destination bucket name, select the S3 bucket pgp-blog-s3-bucketXXXXXXX created by the CloudFormation template.

2.6. As Destination key prefix for our archived files, we choose: archived-files/${transfer:UserName}/${transfer:UploadDate}/

2.7. Select Next, review your selections, and choose Create step.

3. Create the “delete” step.

As the last step, we delete the original file in the user’s home directory.

3.1. In the Nominal steps section, choose Add step.

3.2. On the Choose step type page, select Delete file, and then Next.

3.3. Under Configure parameters, choose a Step name and select Delete the original source file. For this example, we choose “delete” as Step name.

3.4. As File location select Delete the original source file.

3.5. Review your selections, and choose Create step.

At this point your Create workflow page should look similar to the following image:

Configure the Exception handler

If the document received is unencrypted, or the document was encrypted with the incorrect public key, then the decrypt file step generates an exception and the steps defined in the Exception handlers are executed. For this post, in the Exception handlers, we add a custom file-processing step invoking the Lambda function deployed by our CloudFormation template. The name of the function is pgp-blog-exceptionXXXXXXXX. The Lambda function moves the file to a quarantine folder, and then a notification is sent to the operator via Amazon SNS that an error has occurred in the managed workflow.

1. In the Exception handlers section of the Create Workflow page, select Add step.
2. In the Choose step type page, choose Custom-file-processing-step, then choose Next.
3. For Step name, enter a name of your choice and remember not to add spaces! For your Target choose the Lambda function pgp-blog-exception-XXXXXXXX. Set the Timeout to 30 seconds, and choose Next.
4. Review your selections, and choose Create step.
5. In the Create workflow page, at the bottom right of the page, choose Create workflow.

Associate the workflow with the Transfer Family server

Now that we have created our managed workflow, we must associate it with our server.

1. On the left of the screen, select
2. Select the server created by the CloudFormation template. The ID of the Transfer Family server created by CloudFormation can be found in the Outputs section of your CloudFormation stack.
3. Scroll down until you get to Additional detail, and select Edit.
4. In the Managed workflows section, select Workflow for complete file uploads, and choose the managed workflow that we just finished creating.
5. In Managed workflows execution role, choose the IAM role created by the CloudFormation template. The name of the IAM role is PGPblogWorkflowXXXXXXXX.
6. At the end of the page, choose Save.

Add an Admin Alert

If a file comes in unencrypted or has an error during the decryption managed workflow step, then we alert our system admin. To do this, we subscribe to the SNS topic created for us by CloudFormation.

1. Navigate to Amazon SNS, and select Topics.
2. You should see an already created topic called PGPblogSNStopicXXXXXXXX. Select this topic and choose Create subscription.
3. On the Create subscription page, leave Topic ARN as default, select Email as Protocol, enter your email address in the Endpoint textbox, and finally choose Create subscription.

Within a few minutes you should receive an email from AWS Notifications containing a link to confirm your email subscription.

Let’s test it all out!

Now we are ready to test out our managed workflow by uploading files to our Transfer Family server. For our post, we upload an encrypted file and watch the workflow execute. Then, we upload an unencrypted file to see our exception handler in action.

Upload an encrypted file

1. Create a text file by executing the following command:

echo "This is just some text." > test-file.txt

2. Now we encrypt the file using the PGP key we previously generated. The following command creates the encrypted file test-file.txt.gpg.

gpg --encrypt -r  testuser@example.com --openpgp test-file.txt

3. Log in to the server by running the following command:

sftp testuser@<insert your Transfer Family endpoint here>

4. Upload the encrypted file test-file.txt.gpg by running the following command:

put test-file.txt.gpg

5. Go to your S3 bucket pgp-blog-s3-bucketXXXXXXXX to see the uploaded file. We expect to find the decrypted file test-file.txt under the “decrypted-files” prefix, and the original encrypted file test-file.txt.gpg under the “archived-files” prefix. You can also check the CloudWatch logs to see the completed process. Note that it may take some time for all of the data to populate in CloudWatch logs.

Upload an unencrypted file

1. Upload another file, this time an unencrypted file. Once you are logged in to your SFTP server, upload the file we created in the previous step by issuing the following command:

put test-file.txt

2. Go back to your S3 bucket pgp-blog-s3-bucketXXXXXXXX to see the uploaded file. We expect to find the file under the “quarantine-files” prefix. You can also check the CloudWatch logs to see where it logs an exception. Note that it may take some time for all of the data to populate in CloudWatch logs.
3. Check your inbox. You should have received an email from AWS Notifications notifying you of the failed decryption workflow.

Cleaning up

You created several components that may incur costs. To avoid future charges, remove the resources with the following steps:

1. Delete the S3 bucket’s results prefix created after you uploaded files from the sftp server. Use caution in this step. Unless you are using versioning on your S3 bucket, deleting S3 objects cannot be undone.
2. Delete the secret you created in Secret’s Manager. In the Secret details section, choose Actions, and then choose Delete secret.
3. Remove the CloudFormation stack, by selecting the stack on the AWS CloudFormation console, choosing Delete and confirm.

Conclusion

In this post, we walked you through a scenario of uploading files to a Transfer Family server and had those files automatically decrypted upon arrival. We started by creating a PGP key pair and storing it in Secrets Manager along with user configuration details. We created a Transfer Family managed workflow to automatically decrypt incoming encrypted files and store them in Amazon S3. An exception handler was used to handle errors during the decryption process. These files were quarantined in a separate folder and a notification was sent alerting the server administrator of the error.

Pretty Good Privacy (PGP) encryption provides an additional layer of security when exchanging files containing sensitive data, such as health data, financial records, or Personal Identifiable Information (PII). One of the main challenges that customers previously faced using PGP-based encryption was the added layer of operational complexity associated with decrypting files. Now, with Transfer Family’s built in support for automatic PGP decryption of files, that layer of complexity is gone. To learn more about Transfer Family, visit our documentation and product page.

We hope you have enjoyed our post, happy building!