AWS Storage Blog

Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager

UPDATE: An updated version of this post was published on 11/5/2020. Please refer to that post for the most up-to-date content.

Last year at re:Invent we launched AWS Transfer for SFTP (AWS SFTP), a fully managed service that makes it easy to migrate your file transfer workflows to AWS, without changing applications or clients. You can use the service to upload and download files over SFTP directly in and out of Amazon S3. We’ve seen use of the service across a broad range of industry verticals such as financial services, healthcare, retail, and telecommunications. The service supports two ways of managing your end users. You can store your users’ identities within the service, or plugin an existing identity provider of your choice. When you use the service to store your users’ identities, you can enable SSH (Secure Shell) keys for end-user authentication, but what if you need the more traditional password-based authentication or a mix of both?

Thankfully, AWS SFTP supports password authentication when you plug in an identity provider to authenticate and authorize your users. This mode supports both forms of authentication – passwords and SSH keys.

In this post, I will show you how to use password authentication with AWS Transfer for SFTP and dynamic role allocation for access to Amazon S3. This is done by integrating with a custom identity data provider (IdP) and in this example, I demonstrate using AWS Secrets Manager as the IdP.

Plugging in your identity provider

By default, a new AWS SFTP server uses its internal user directory for SSH key-based authentication. You can change it to use an IdP of your choice. To do this, specify --identity-provider-type API_GATEWAY with an API Gateway endpoint to map access to the custom authentication provider.

With this configuration, the following workflow kicks in to authenticate and authorize your users:

The steps are as follows:

  1. A user attempts to log in, supplying either a user name and password, or a user name and private SSH key (stored local to their disk).
  2. The service passes these credentials to the API Gateway endpoint you provided when you created the server. The API Gateway is integrated with an AWS Lambda function. If the user does not provide a password, it is assumed that they are using SSH key-based authentication.
  3. The Lambda function queries the custom authentication provider (which can be any datastore, and in this case AWS Secrets Manager) using the user provided credentials.
  4. Secrets Manager returns the key-value pairs associated with the user or secret. This contains the user’s stored password, the IAM role mapping for the user, and any public SSH key information (if you allow SSH key-based authentication for the user).
  5. The login is validated when the Lambda function validates the password match or you have an SSH key value present in Secrets Manager. If you have a valid user login, then the Lambda function constructs an HTTP 200 response with the remaining key-value pairs. If you do not have a valid user login, then return an empty HTTP 200 response.

The API Gateway endpoint provides a single method with a resource path:

/servers/serverId/users/username/config

The serverId and username values come from the RESTful resource path. For security, the user’s password is passed through a password header in the request.

The payload returned in the API Gateway method response consists of the following values:

  • Role: ARN of the IAM Role that contains the policy used to provide the user access to your S3 bucket.
  • (Optional) HomeDirectory: The home directory (in the AWS SFTP case, the S3 bucket location) into which the user is dropped on login.
  • (Optional) Policy: An STS.AssumeRole scope-down policy blob that further restricts the IAM role based on certain parameters.
  • (Optional) PublicKeys: If no password was provided (SSH key-based authentication), then the public SSH key associated with the user is returned.

In this example we’re only using Role and HomeDirectory.

So where should you store your secrets?

Now that you understand how custom authentication for AWS SFTP works, here’s a secure data store with a RESTful API in which to store your user data: Secrets Manager.

Create user entries along with your custom attributes (password, IAM role, and HomeDirectory value) in an encrypted store with Secrets Manager. You can provide granular access to only those who require it.

You can’t directly connect AWS SFTP to Secrets Manager today, so you will use a Lambda function that provides the logic to connect them. This Lambda function is responsible for validating the user credentials against the one stored, and return access information.

Start building

Create the setup in the earlier diagram, which includes an API Gateway endpoint with a single method.  AWS SFTP invokes this method when your user’s SFTP client sends an authentication request. This method invokes a Lambda function that in turn queries Secrets Manager for user configuration data, validates that the password matches, and returns the remaining information for your users’ access to S3.

CloudFormation template

All of the API Gateway and Lambda resources to set up the integration are deployed in aws-transfer-custom-idp-secrets-manager-apig CloudFormation template. Download it and create the stack.

To get up and running quickly, skip the explanation and jump to Create the SFTP server.

(Update 10/17/19: I’ve updated the template referenced in this post to support “chroot” using the Logical directories features announced in September 2019, as well as optionally launching the AWS SFTP endpoint which should simplify your setup and reduce configuration errors. Please visit my follow on post titled “Simply Your AWS SFTP Structure with Chroot and Logical Directories” to learn more!)

API Gateway API

This is the glue between AWS SFTP and Secrets Manager. The stack creates a Get resource to match the required API path /servers/serverId/users/username/config.

Method Request passes the following values through:

  • Query string: serverId, username
  • Headers: password

Integration Request maps the following variables into the Lambda function input parameters using the mapping template:

  • username
  • password
  • serverId

Integration Response passes through the results returned by the Lambda function.

Finally, Method Response returns a structured HTTP 200 response with the Lambda function output.

Lambda function

The Lambda function is a relatively simple piece of Python code. It takes the input parameters from the incoming API Gateway request and looks up the user name (in the format SFTP/Username) as a secret in Secrets Manager.

  • If the password is passed through to the function by AWS SFTP, authenticate the user by password so that it is validated against the password stored within the secret.
  • If the password is blank, then authenticate the user by SSH key and pass the stored public keys back to AWS SFTP.

In either case, also pass back the remaining parameters from the secret (Role, Home Directory, and so on).

There is also a Lambda environment variable that defines the Region in which Secret Manager should be accessed. This defaults to the same Region into which the CloudFormation stack was deployed.

IAM roles

  • API Logging Role: TransferApiCloudWatchLogsRole – Allows the API Gateway endpoint to log to CloudWatch Logs.
  • API Gateway Access Role: TransferApiInvokerAssumeRole – Allows AWS SFTP to call the API Gateway endpoint, which means only services that you provide access to can use this API Gateway endpoint.
  • Lambda Role: TransferLambdaExecutionRole – Allows the Lambda function to execute, and provides read-only access to Secrets Manager for secrets with the prefix SFTP/*.

Create the SFTP server

If you don’t already have an IAM role that you can map to your AWS SFTP users, create one. This allows your users to access your S3 bucket for uploads or downloads. For more information, see Create IAM Policies and Roles for SFTP.

If you want your AWS SFTP server to be able to write to CloudWatch you will also need to create a role for this (e.g TransferCloudWatchLogsRole). You can use the TransferApiCloudWatchLogsRole provided in the stack as a starting point for this.

Next, create an AWS SFTP endpoint using your new authentication provider. Provide the API Gateway endpoint and the IAM role that authorizes the service to invoke this API Gateway endpoint. Get these values from the CloudFormation stack output.

Feel free to use the console or the AWS CLI. The AWS CLI command is shown as follows:

URL=https://APIGATEWAYID.execute-api.us-east-1.amazonaws.com/prod

INVOCATION_ROLE=arn:aws:iam::xxxxxxxxxxxx:role/xxxxx-TransferIdentityProviderRole-xxxxx 

aws --region us-east-1 transfer create-server --identity-provider-type API_GATEWAY --identity-provider-details "Url=${URL},InvocationRole=${INVOCATION_ROLE}"

If you chose to create a logging role, add --logging-role=arn:xxxxx to the command or select the role in the AWS SFTP console.

Now, set up some users in Secrets Manager:

  1. In the Secrets Manager console, create a new secret by choosing Store a new secret.
  2. Choose Other type of secret.
  3. Create the following key-value pairs. The key names are case-sensitive.
Key - Password
Value - mySup3rS3cr3tPa55w0rd
 
Key - Role
Value - arn:aws:iam::xxxxxxxxxxxx:role/AWSTransferAccessRole
Explanation: Use one of the Role ARNs you created for AWS SFTP users earlier. This will define what access the user has to S3
 
Optional:
Key - HomeDirectory
Value: /myawstransferbucket/home/jess
Explanation: The path to the users home directory 
 
Key - PublicKey
Value - ssh rsa public-key
Explanation: Public portion of SSH key pair data

Save the secret in the following format: SFTP/username.

Ready to test

The AWS SFTP API provides a function to test whether the external authentication is working as expected. Swap in your SFTP server-id value, plus the user name and password that you entered in Secrets Manager:

aws transfer test-identity-provider --server-id "s-xxxxxxxxxx" --user-name charlie --user-password password

If all is working and you have passed through the correct user details, you should get the role information that you provided in Secrets Manager in the response field.

$ aws –-region="us-east-1" transfer test-identity-provider –-server-id s-52ea3fbf819a47f69 –-user-name jess –-user-password mySup3rS3cr3tPa55w0rd
 
{
    "Response": "{\"Role\": \"arn:aws:iam::012345678901:role/AWSTransferAccessRole"}",
    "StatusCode": 200,
    "Message": "",
    "Url": "https://d0vmskd8g7.execute-api.us-east-1.amazonaws.com/prod/servers/s-52ea3fb819a47f69/users/jess/config"
}

If the user is not authenticated (wrong user name or password), the test should return an error in the message field. If you don’t get the expected response at this stage, check the logs.

Finally, go ahead and try logging in with your external user details:

sftp username@s-xxxxxxxxxxxx.server.transfer.region.amazonaws.com

Cleaning up

If you are done with the resources—including the AWS SFTP server that you deployed today, don’t forget to clean up and check for any permissions that are no longer required.

Summary

That’s it! You now have access to a fully managed, highly available SFTP server that can authenticate your users using passwords, and use dynamic role allocation provided by a custom authentication provider for SFTP access to Amazon S3.