Simplify Active Directory authentication with a custom identity provider for AWS Transfer Family

The secure exchange of files business-to-business is a common and important business practice in highly regulated industries. As organizations grow, maintaining separate user credentials for various platforms, such as file transfer services, can become increasingly complex and time-consuming. As customers build out Managed File Transfer (MFT) environments, often they end up having to create separate, or branched, identity solutions for authenticating users, resulting in more complexity.

Custom identity providers (IdPs) for AWS Transfer Family offer a powerful solution to the challenge of managing user accounts across multiple enterprise systems. To offer users maximum flexibility to integrate their own providers and advanced authorization logic for secure authentication for file transfer, Transfer Family provides custom IdP examples. Historically, these examples have been published in various posts and within documentation. With the AWS Transfer custom IdP solution, we have consolidated our most popular custom IdPs, such as Active Directory (AD) and Okta, into a standardized modular package.

In this post, we introduce a modular custom IdP solution for Transfer Family and demonstrate how to deploy and configure it to authenticate users in an AD domain. We walk you through the initial configuration of the custom IdP solution, including configuring the custom IdP itself and end user accesses for specific data. This custom identity provider solution can help you simplify authentication and authorization across disparate enterprise systems by consolidating identity providers like Active Directory into a modular, serverless architecture for secure managed file transfer.

Solution overview

This IdP solution separates authentication and authorization logic, offering a flexible and easy-to-maintain foundation for various use cases. The solution provides these key features:

• An AWS Serverless Application Model (AWS SAM) template that provisions the required resources. Optionally, deploy and configure AWS API Gateway to incorporate AWS WAF.
• An Amazon DynamoDB schema to store configuration metadata about users and IdPs, including user session settings such as HomeDirectoryDetails, Role, and Policy.
• A modular approach that enables you to add new IdPs to the solution in the future as modules, such as the user’s own modules.
• Support for the following IdPs: LDAP (such as Microsoft AD), Okta, Public and Private Key, and AWS Secrets Manager.
• Support for multiple IdPs connected to a single Transfer Family Server and multiple Transfer Family servers using the same deployment of the solution.
• Built-in IP allow-list checking, such as IP allow lists that can optionally be configured on a per-user or per-IdP basis.
• Detailed logging with configurable log-level and tracing support to aide in troubleshooting.

The solution deploys an AWS Lambda function along with an Amazon DynamoDB database to store configuration metadata about users and IdPs. You can plug in different IdP modules (e.g., LDAP, Okta, public/private keys) into the solution to handle authentication against various identity sources. This modular approach enables the addition of new IdPs in the future and provides the ability to develop custom modules. The user records in the DynamoDB table map usernames to specific IdPs and store per-user settings like home directory details, roles, and POSIX profiles. When a user attempts to connect to the AWS Transfer Family server, the custom IdP Lambda function authenticates the user against the configured IdP module, retrieves the user-specific session settings from DynamoDB, and provisions those settings for the Transfer Family session.

Figure 1: Custom identity provider solution architecture

Learn more about the solution and its architecture by reviewing the README in the new Toolkit for Transfer Family code repository.

Walkthrough

Here we demonstrate a use case where an organization requires AD authentication for their Transfer Family server. The organization needs the flexibility to assign each user unique Transfer Family session settings, such as a Role, HomeDirectoryDetails, and PosixProfile. We cover the following in this walkthrough:

1. Deploying custom IdP and AWS Managed Active Directory:

• • • Setting up a custom Identity Provider (IdP)
    • Provisioning all the necessary resources required for this solution (e.g. Microsoft AD)

2. Define an IdP:

• • • Configuring the IdP provider settings

3. Define a user:

• • • Managing user records within the IdP

4. Test the IdP:

• • • Verifying the deployed solution’s functionality

5. Adding public keys to a user:

• • • Using a public key within a user record

The following steps use example templates from the repository to configure resources.

1. Deploying custom IdP and AWS Managed Active Directory

We deploy the following resources in this step:

• Transfer Family server
• Transfer Family custom IdP Solution
• AWS Managed Microsoft AD
• Amazon Virtual Private Cloud (VPC), subnets, NAT gateways, and internet gateway (IGW)

1.1. Log in to the AWS account in which you want to deploy the solution, switch to the AWS Region in which you plan to run Transfer Family, and start an AWS CloudShell session, as shown in the following image.

Figure 2: CloudShell in AWS Management Console

1.2. Run the following command to clone the solution’s repository into your CloudShell environment.

cd ~
git clone https://github.com/aws-samples/toolkit-for-aws-transfer-family.git

1.3. Run the following command to run the build script, which downloads the package dependencies and generates artifacts used in the solution. Monitor the execution and verify that the script completes successfully. This should take 1-2 minutes.

cd ~/toolkit-for-aws-transfer-family/solutions/custom-idp
 ./build.sh

1.4. Begin the AWS SAM deployment by using the following command:

sam deploy --guided --template ./examples/activedirectory/one-click-template.yaml --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM --disable-rollback

1.4. When prompted, provide the following information:

• • • Stack name: transferidp
    • AWS region: us-east-1
    • DirectoryDNSName: corp.demosftp.com
    • DirectoryAdminPassword: <Type an admin password here>
    • DirectoryNewUsername: jsmith
    • DirectoryNewUserPassword: <Type a user password here>
    • DirectoryBIOSName: CORP
    • DirectorySearchBase: OU=Users,OU=corp,DC=corp,DC=demosftp,DC=com
    • SecretsManagerPermissions: true
    • UserNameDelimiter: @
    • LogLevel: INFO
    • ProvisionApi: false
    • EnableTracing: false
    • UsersTableName: empty
    • IdentityProvidersTableName: empty
    • Confirm changes before deploy: N
    • Allow SAM CLI IAM role creation: Y
    • Disable rollback: Y
    • Save arguments to configuration file: Y
    • SAM configuration file: leave default
    • SAM configuration environment: leave default

1.5. Once the deployment completes successfully, copy/paste the CloudFormation outputs from deployed stack to a text editor and save them for future use. You need this additional information listed in later configuration steps. You can also open the AWS CloudFormation console in a separate tab to monitor the progress.

The following are descriptions of the outputs:

2. Define an IdP

Once the Transfer Family custom IdP solution is deployed, you must define one or more IdPs in the identity_providers DynamoDB table. Each record stores an IdP configuration. For complete details on IdP modules, settings, and examples, see the IdP modules. To get started, we define an IdP that uses the ldap module to connect to the Managed AD domain deployed in the previous section.

2.1. In the AWS Management Console, navigate to the DynamoDB Item Explorer and select the providers table (such as Table name [StackName]-CustomIdP-XXXX_identity_providers) under the select a table drop down. Select the Create item button.

Figure 3: DynamoDB table for the custom IdP

2.2. In the Create item screen, select JSON view, and copy and paste the following record, replacing [ADPrimaryDNS] with the value of ADPrimaryDNS from the CloudFormation outputs in Step 1.5:

{
  "provider": {
    "S": "MicrosoftAD"
  },
  "config": {
    "M": {
      "attributes": {
        "L": []
      },
      "search_base": {
        "S": "OU=Users,OU=corp,DC=corp,DC=demosftp,DC=com"
      },
      "ssl": {
        "BOOL": false
      },
      "ssl_verify": {
        "BOOL": false
      },
      "port": {
        "S": "389"
      },
      "server": {
        "S": "[ADPrimaryDNS]"
      }
    }
  },
  "module": {
    "S": "ldap"
  }
}

2.3. Select the Form button to switch back to form view. The attributes of the record are displayed as shown in the following screenshot once you expand the config section.

Figure 4: Creating the IdP record in DynamoDB

• • • The provider attribute contains the name of the IdP. In this case, it is MicrosoftAD, but it could be more meaningful, such as enterprise-AD or example.com if it were an external IdP.
    • The module attribute contains the name of the IdP module to use, in this case ldap.
    • The config is an attribute map that is used for storing IdP configuration details. You can find a description of each parameter in the ldap module documentation.

2.4. After reviewing, select Create item. The first IdP has now been defined. Next, we begin defining users.

3. Define a user

Once the IdP is defined, a user must be defined. Each user record contains the settings that are used for a Transfer Family session. Each record also maps the username to a given IdP. In this section, we create a user record and map it to the ldap IdP created in the previous section.

3.1. An AWS Identity and Access Management (IAM) role is created in the CloudFormation template. The TransferUserRole parameter is in the outputs of Step 1.5.

3.2. In the Console, navigate to the DynamoDB Item Explorer, and select the users table (such as Table name [StackName]-CustomIdP-XXXX_users) under the Select a table drop down. Select the Create item button.

Figure 5: DynamoDB table for the users

3.3. In the Create item screen, select JSON view, and copy and paste the following record, replacing [SFTPServerS3Bucket] and [TransferUserRole] with the matching CloudFormation outputs in Step 1.5:

{
  "user": {
    "S": "jsmith"
  },
  "identity_provider_key": {
    "S": "MicrosoftAD"
  },
 "config": {
   "M": {
     "HomeDirectoryDetails": {
       "L": [
         {
           "M": {
             "Entry": {
               "S": "/"
             },
             "Target": {
               "S": "/[SFTPServerS3Bucket]/${transfer:UserName}"
             }
           }
         }
       ]
     },
     "HomeDirectoryType": {
       "S": "LOGICAL"
     },
     "PosixProfile": {
       "M": {
         "Gid": {
           "S": "1000"
         },
         "Uid": {
           "S": "1000"
         }
       }
     },
     "Role": {
       "S": "[TransferUserRole]"
     }
   }
 },
 "ipv4_allow_list": {
   "SS": [
     "0.0.0.0/0"
   ]
 }
}

3.4. Select the Form button to switch back to Form view and expand the nested attributes. The attributes of the record are displayed as shown in the following screenshot. The following details are from the various fields:

• • • The user key contains the username (such as jsmith) that are passed to Transfer Family during authentication.
    • The identity_provider_key attribute contains the IdP name from the [StackName]_identity_providers table, which we created in the previous section.
    • The ipv4_allow_list attribute is a list of remote IP CIDRs that are allowed to authenticate as the user. This is an optional attribute and by default the remote IPs are allowed to authenticate as the user.
    • The config attribute is a mapping of the user’s session settings. Its values follow the same format as those found in the AWS Lambda Values Section of the custom IdP documentation. This includes the HomeDirectoryType, HomeDirectoryDetails (for logical directory mappings), and PosixProfile. Note that HomeDirectoryDetails can have both Amazon S3 and Amazon Elastic File System (EFS) targets, in the event that you want to connect users to both S3 and EFS Transfer Family servers.

Figure 6: User setup – user record

3.5. After reviewing, select Create item. The first user, jsmith, has been created and mapped to the MicrosoftAD identity provider.

4. Test the IdP

To test the IdP and user created in the previous sections, use an SFTP client to connect to the Transfer Family server. For example, open a CloudShell session and enter the following command to connect, replacing [TransferServerEndpoint] with the CloudFormation output from step 1.5.:

sftp jsmith@[TransferServerEndpoint]

The following is an example of a successful connection and file/directory listing:

Figure 7: A screenshot of a successful provider testing

To view the provider logs, open Amazon CloudWatch Logs and select the log group /aws/lambda/[StackName]_awstransfer_idp. The log streams show the log in event and details about the user and IdP evaluation. If there was a failure, then you should also see error messages and exceptions in this log. The following is a screenshot showing the logging in near real-time using CloudWatch Live Tail.

Figure 8: Viewing the provider logs in CloudWatch

5. Adding public keys to a user

Some organizations would like to support both AD and public key authentication for users. The solution allows you to add public keys to the user record, allowing public key and password authentication to be used together. The following are the steps to do this with an existing public key:

5.1. Open a CloudShell session and generate a public/private keypair using this command:

ssh-keygen -t rsa -N '' -f transfer_key

5.2. Retrieve the public key using the cat command and copy the output starting with ssh-rsa to your clipboard.

cat transfer_key.pub

5.3. In the Console, navigate to the DynamoDB Item Explorer, and select the users table (such as Table name [StackName]-CustomIdP-XXXX_users) under the Select a table drop down. Select the jsmith record in the users table to edit the item.

5.4. In the Edit item screen, expand the config Select Insert a field under the config attribute, and select String set.

5.5. For Attribute name, enter PublicKeys. In the Empty value field, paste your public key. Also note that multiple public keys can be added for each user by adding more values to this field.

5.6. Select Save to update the user record. The record should be similar to Figure 9, with the PublicKeys field added containing the public key for access.

5.7. Use the command similar to the following to verify that you can perform public key authentication with the user.

sftp -i transfer_key jsmith@[TransferServerEndpoint]

Note that you can also configure the Transfer Family server to use multi-method authentication to enforce Password and Key authentication.

Cleaning up

To clean up the deployment, follow these steps:

1. Run the following command with the stack-name that you used in Step 1.5 to deploy the solution.

sam delete –-stack-name <stack-name>

2. There are three resources that require manual deletion, because by default they are retained to preserve user/provider data that you may have entered.

• • • Two DynamoDB tables:
      • • <stack-name>-CustomIdp-<xxxxx>_identity_providers
        • <stack-name>-CustomIdp-<xxxxx>_users

To delete these tables, go to the DynamoDB tables console page, select the checkboxes in front of the tables, and then select the Delete button in the upper corner.

• • • AWS Lambda Layer:
      • <stack-name>-CustomIdP-<xxxx>-handler-layer

To delete this layer, go to the Lambda layers console page, select the Lambda layer you want to delete, and then select the Delete button in the upper corner.

Conclusion

In this post, we introduced the AWS Transfer Family custom identity provider (IdP) solution for managing users and demonstrated how to deploy and use the solution with Active Directory (AD). The combination of pre-built identity provider modules and flexibility to apply per-user settings to AWS Transfer Family sessions makes it a compelling option for customers implementing custom IdPs.

Thank you for reading this post. You can get started with deploying this Transfer Family IdP solution today. The AWS Serverless Application Model (AWS SAM) template used in this post consists of three nested AWS CloudFormation templates to deploy the Transfer server, custom IdP, and AWS Managed AD. You can reuse these templates separately and independently to fit your deployment needs. Take a moment to review the full solution on GitHub, test it with other scenarios, and provide feedback by submitting an issue in the repository.