Transfer customer managed SSE-KMS encrypted objects across AWS accounts and Regions using AWS DataSync

Update (7/12/2024): Post updated to clarify that you must use a fully qualified Amazon Resource Name (ARN) when specifying your customer managed KMS keys.

Some organizations have requirements to manage their own data encryption keys, both in general and during data transfer processes. In addition, when considering data transfer solutions (not just for encrypted data), organizations must think about factors such as preventing unauthorized access during transfer and storage, transfer efficiency, data integrity, and monitoring mechanisms to make sure the transfer is functioning correctly.

AWS DataSync allows you to move your file and object data between on premises and AWS, between AWS Storage services, and between AWS and other public clouds. Users often have requirements to transfer encrypted data between Amazon S3 buckets that are in different AWS Regions and accounts. With DataSync, you can transfer existing SSE (server-side encryption) encrypted S3 objects across AWS accounts and Regions and also consolidate data to a single S3 bucket. Another option commonly consider for this use case is S3 Replication. Although both are good choices, DataSync is often used in cases where you have existing data in the source bucket and you don’t want to enable bucket versioning, or in cases in which you want to change the encryption method to be different from what it was at the source (for example, from S3 managed keys, SSE-S3, to an AWS Key Management Service (AWS KMS) customer managed key). A discussion of both options is available in the post, “Considering four different replication options for data in Amazon S3.” However, if DataSync fits your use case, read on!

In this post, I walk through configuring DataSync, including creating AWS Identity and Access Management (IAM) roles and updating AWS KMS key policies, to transfer SSE-KMS encrypted data between S3 buckets (in different AWS accounts and Regions) that use different customer managed AWS KMS keys. This allows you to securely and efficiently transfer data while still maintaining control over the data encryption keys used to store your data.

Solution overview

Before beginning, let’s review the built-in server-side encryption options for S3 buckets:

1. Server-side encryption with Amazon S3 managed keys (SSE-S3). All new object uploads to Amazon S3 buckets are encrypted by default with server-side encryption with Amazon S3 managed keys (SSE-S3).
2. Server-side encryption with AWS KMS keys (SSE-KMS) and dual-layer server-side encryption with AWS KMS keys (DSSE-KMS) have two encryption key options:
   • AWS managed keys with a key alias of “aws/s3”.
   • Customer managed keys with a key alias that you specify when you create the key.
3. Server-side encryption with customer-provided keys (SSE-C) (currently not supported by DataSync).

Although the different encryption options might appear to introduce some complexity when trying to transfer data with DataSync, especially when the source and destination buckets have different types of encryptions, there is a simple rule you can follow. If DataSync can access both buckets and their keys by using an IAM role, then DataSync can access the data that you want to transfer. Let’s look at what this means when using the encryption types mentioned previously:

1. SSE-S3: The keys are managed by Amazon S3 and transparent to you and DataSync. Because the DataSync IAM role can be granted access to Amazon S3, it can read or write the data in either the source or destination account.
2. AWS managed key (aws/s3): These keys have policies that can’t be modified. As a result, you can’t change a key policy to permit access from a cross account IAM role. The DataSync IAM role can only access the encryption key in the same account where the DataSync task is running. The source or destination can use this encryption method, but not both, and the DataSync task must run in the same account that is using this method.
3. Customer managed key: You can edit the key policy in the source or destination account to grant access to the DataSync IAM role so that DataSync can access bucket data.

This post focuses on using customer managed keys, but it can be used with other encryption options when the previously mentioned rule is applied.

DataSync manages the transfer of data between supported AWS Storage services without requiring additional customer managed infrastructure. You simply define the source and destination locations within DataSync. Then create a task to initiate the transfer from source to destination. When transferring between AWS Storage services (whether in the same Region or across AWS Regions), your data remains in the AWS network and doesn’t traverse the public internet. Additionally, DataSync encrypts data transferred between locations with TLS 1.3.

Figure 1: DataSync access requirements for data transfers across AWS accounts and AWS Regions

DataSync locations describes where you’re transferring data from or to. These locations require an IAM role to access your data. As shown in the preceding figure, the source and destination DataSync locations are created in the source account in the same Region as the corresponding S3 buckets. The DataSync source IAM role needs access to the source S3 bucket and AWS KMS key policy used to encrypt the objects. Similarly, the DataSync destination IAM role needs access to the destination S3 bucket and AWS KMS key policy to encrypt the objects at the destination location.

This solution builds on the tutorial, Transferring from S3 to S3 in another account, which guides you through the setup of how to transfer Amazon S3 data across accounts and AWS Regions. The buckets can be encrypted with the SSE-S3 in that solution, and this post provides the additional steps to configure customer-managed SSE-KMS keys.

Prerequisites

Before you begin the walkthrough, you must have two AWS accounts. If you don’t have AWS accounts already, then you can sign up here. You should also have intermediate knowledge of DataSync, Amazon S3, AWS KMS, and IAM.

This solution assumes you already have the following in place:

• A pair of S3 buckets in different AWS accounts and AWS Regions.
• AWS KMS customer managed keys created in each AWS account and associated as the SSE-KMS default encryption on the respective S3 buckets. Since you’re working with cross-account buckets, the buckets must use the fully qualified ARN of their respective KMS key (not just a key alias or ID).
• An AWS user account with permission to DataSync, Amazon S3, AWS KMS, and IAM in the source and destination accounts.

Walkthrough

In this example, I have a source S3 bucket that’s in one AWS account and AWS Region. I also have a destination bucket that’s in a different AWS account and AWS Region. Objects in the source S3 bucket are encrypted using a customer-managed AWS KMS key. I want to transfer objects from the source S3 bucket to the destination S3 bucket. The destination bucket is empty and has encryption enabled using a customer-managed KMS key. The following steps outline the process:

1. Create DataSync IAM roles to allow DataSync to transfer data on your behalf
2. Update the KMS key policy in your source AWS account and AWS Region used for the source S3 bucket
3. Update DataSync destination IAM role in your source account with permission to access the KMS key in your destination account
4. Update the KMS key policy in your destination AWS account and AWS Region for the destination S3 bucket
5. Create the DataSync locations and task to transfer data

1.Create DataSync IAM roles to allow DataSync to transfer data on your behalf

In this step you create a DataSync source and destination IAM roles in the source account to transfer data between the S3 buckets.

When you create a transfer location for a bucket, DataSync can automatically create and assume a role that normally has the correct permissions to access that bucket. Since you’re transferring across accounts and the S3 buckets are encrypted with customer-managed SSE-KMS keys, you must create or update the roles manually.

1. Create the DataSync source IAM role in the source account.

a. Open the IAM console.

b. In the left navigation pane, under Access management, choose Roles, and then choose Create role.

c. On the Select trusted entity page, for Trusted entity type, choose AWS service.

d. For Use case, choose DataSync in the dropdown list, and select DataSync – S3 Location. Choose Next.

e. On the Add permissions page, the AmazonS3FullAccess policy is automatically selected. Choose Next.

f. Give your role a name and choose Create role.

2. Attach a custom IAM policy to the IAM role.

a. On the Roles page of the IAM console, search for the IAM role that you just created and choose its name.

b. To narrow the policy scope, remove the AmazonS3FullAccess IAM policy by selecting the check box and choose Remove.

c. Attach a custom IAM policy by choosing Add permissions and then Create inline policy.

d. Choose the JSON tab and paste the following JSON into the policy editor:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Resource": "YourS3BucketArn"
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectTagging",
                "s3:PutObjectTagging",
                "s3:PutObject"
              ],
            "Effect": "Allow",
            "Resource": "YourS3BucketArn/*"
        }
    ]
}

e. Replace the “YourS3BucketArn” with the source S3 bucket ARN in your source account.

3. While still in your source account, repeat the previous steps to create the DataSync destination IAM role. This time, replace the “YourS3BucketArn” with the destination S3 bucket ARN in your destination account in the IAM policy.

2. Update the KMS key policy in your source AWS account and AWS Region used for the source S3 bucket

This grants the DataSync source IAM role in your source account access to the KMS key for the source S3 bucket in your source account.

1. Open the AWS KMS console.

2. Choose the customer managed KMS key that you have configured for the source S3 bucket.

Figure 2: Source S3 bucket KMS key

3. Scroll down to Key users and choose Add.

Figure 3: Key users/roles allowed to use the KMS key

4. Filter the list by entering the source DataSync IAM role that you previously created into the search box, select the role, and choose Add.

Figure 4: Grant source DataSync role access to the KMS key

3. Update the DataSync destination IAM role in your source account with permission to access the KMS key in your destination account

You must update your DataSync destination IAM role in your source account so that it can use your destination AWS KMS key.

1. Log in to your source account.

2. In the AWS Console, navigate to IAM.

3. Choose Roles.

4. Search for the DataSync destination IAM role by entering the role name in the search box.

Figure 5: DataSync destination IAM role in your source account

5. Choose your role.

6. Choose Add permissions and choose Create inline policy.

Figure 6: IAM Create inline policy

a. Choose JSON tab to switch to the JSON editor.

Figure 7: JSON tab

b. Replace the default statements with the following policy to add only the required KMS key permissions following a least-privileged model.

{
	"Version": "2012-10-17",
	"Statement": [
{
"Sid": "AllowUseOfKeyInAccount111122223333",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:destinationregion:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}
	]
}

c. Replace “AllowUseOfKeyInAccount111122223333” with your destination account information.

d. Replace “arn:aws:kms:destinationregion:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab” with the ARN of the KMS key in your destination AW account and AWS Region used for the destination S3 bucket encryption. You must specify the key’s fully qualified ARN (you can’t specify just a key ID or alias in a cross-account situation).

e. Choose Review Policy to save the changes.

f. Enter a name for your policy and choose Create Policy to complete the addition of the policy.

Figure 8: Provide name for inline policy

4. Update the KMS key policy in your destination AWS account and AWS Region for the destination S3 bucket

 This grants the DataSync destination IAM role in your source account access to the KMS key that is used with your destination S3 bucket. Allowing cross-account use of a customer managed KMS key requires updating the KMS key policy JSON instead of selecting from a list because the local account is not aware of IAM users or roles in a different account.

1. Log in to your destination account.

a. In the AWS Console, navigate to AWS KMS, then Customer managed keys.

b. Choose the KMS key that you have configured for the destination S3 bucket.

Figure 9: Destination S3 bucket KMS key

c. Scroll down to Key policy and choose Switch to policy view.

Figure 10: Switch to policy view

d. Choose Edit to update the key policy with the DataSync destination role of your source account to access this key.

Figure 11: Edit to update JSON KMS key policy

e. Add the following policy to the key. Replace the “arn:aws:iam::444455556666:role/ExampleRole” role with your DataSync destination IAM role from your source account.

{
    "Sid": "Allow an external account to use this KMS key",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::444455556666:role/ExampleRole"
    },
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
 "kms:DescribeKey"
    ],
    "Resource": "*"
}

f. Choose Save changes to complete the update.

5. Create the DataSync locations and task to transfer data

With the DataSync source and destination IAM roles that you have created in this post, you can now create the DataSync locations and task to transfer data between S3 buckets by following these steps in the tutorial Transferring data between Amazon S3 buckets across AWS accounts.

1. In your destination account, update your S3 bucket policy to include the DataSync destination IAM role in your source account that you created in this post.

2. In your destination account, disable ACLs for your S3 bucket.

3. In your source account, create your DataSync locations.

4. In your source account, create your DataSync task.

Figure 12: DataSync transfer task

Verify data transfer

Once you create the DataSync task, you can verify that DataSync can transfer encrypted objects.

Follow these steps:

1. Upload objects to your S3 bucket in your source AWS account.

2. Start the DataSync task. The task goes through multiple steps, and you can refer to the documentation to understand the status of the different phases of the task.

3. Verify that your data completely transferred to the Amazon S3 destination and encrypted with the destination KMS key. You can verify by checking the server-side encryption settings under the Properties tab of the objects in the destination S3 bucket in the console. You can also verify through the S3 API using the HeadObject API to retrieve the metadata from the object without returning the object itself.

aws s3api head-object --bucket my-bucket --key index.html

Cleaning up

If you are no longer using the resources discussed in the post, I suggest that you clean up the AWS resources. To accomplish this after finishing the proof of concept, clean up/delete the following resources:

• DataSync task
• DataSync source and destination locations
• Disable and schedule deletion of the KMS keys. Be careful deleting KMS keys as this is irreversible and data encrypted with the KMS key becomes unrecoverable.
• Delete the objects in the S3 buckets and delete the S3 buckets
• Delete the IAM roles created in each of the accounts

Conclusion

In this post, I walked through how to transfer SSE-KMS encrypted Amazon S3 objects across accounts and AWS Regions using AWS DataSync, when the source and destination S3 buckets use different customer managed KMS keys. I detailed the steps necessary to configure the AWS KMS key policies, IAM policies, and IAM roles for DataSync locations across accounts and AWS Regions. Additionally, I demonstrated the configuration process to transfer data using DataSync and how to verify data was transferred and encrypted with the specified customer managed key.

AWS DataSync in combination with AWS KMS allows you to securely and efficiently transfer encrypted data, while maintaining control over the encryption keys and meeting compliance requirements. This gives you the benefits of a fully managed transfer service which maintains the security and confidentiality of the data at the source and destination.

To learn more about DataSync, visit AWS DataSync or get started building this architecture in the AWS Management Console. For more use cases when using DataSync, check out our list of posts. For Amazon S3 Replication options, you can check out this post, “Considering four different replication options for data in Amazon S3.”