Protecting your high-performance file systems with Amazon FSx for Lustre

As companies shift high-performance workloads toward cloud solutions, data storage and data protection go side-by-side. Many companies have both internal and external security rules and regulations they must adhere to when storing their data. Amazon FSx for Lustre offers fully managed, scalable file systems for fast-processing workloads, providing secure, shared access to your users.

In this blog, we show you how you can safeguard your data using FSx for Lustre’s encryption feature. This will help you improve security, and limit the likelihood of a breach resulting in data loss. AWS uses the shared responsibility model for safe cloud computing.

Encrypting your FSx for Lustre file systems

FSx for Lustre supports two types of encryptions: encryption-at-rest and encryption-in-transit. However, when you create a FSx for Lustre file system, the encryption of data at-rest automatically activates and uses the XTS-AES-256 block cipher encryption algorithm to encrypt the file system. If you are using a temporary or scratch file system, it encrypts the data at-rest using the unique keys managed by Amazon FSx, and the keys are destroyed after the file system is deleted.

FSx for Lustre persistent file systems let you encrypt the data at-rest either by specifying the customer-managed AWS Key Management Service (AWS KMS) or AWS managed key. By default, persistent file systems use an AWS managed key. FSx for Lustre ensures the data is automatically encrypted before being written to the file system. Similarly, it’s decrypted before being presented to the application. This process results in no coding or application changes required by the customer.

You can see persistent file system encryption details via the AWS Management Console, API, and by running the following command from the Amazon EC2 Lustre client. Scratch file system keys are not displayed via console, API, or through CLI.

$ aws fsx describe-file-systems --region <region>

FSx for Lustre automatically encrypts the Scratch 2 and persistent file systems data-in-transit when they are accessed from EC2 instances that support encryption in-transit. This feature of encryption-in-transit is available in certain regions.

To check the EC2 instance types that support encryption in transit, use the following command:

$ aws ec2 describe-instance-types \
--filters Name=network-info.encryption-in-transit-supported,Values=true \
--query "InstanceTypes[*].[InstanceType]" --output text | sort

FSx for Lustre uses and accepts only symmetric AWS KMS keys. Symmetric keys are used in symmetric encryption, where the same key is used for encryption and decryption. Amazon FSx handles everything related to the encryption and decryption of data-at-rest using AWS KMS.

When you use the default AWS KMS key for Scratch 2 file systems, you will be charged for the usage but not charged for creating and storing an AWS KMS Key. For persistent FSx for Lustre file systems, you can create custom keys also known as customer managed keys, you will be charged to create and store them. The use of customer managed keys helps enable key rotation, configure its key policies and grants permissions for multiple users or services, and choose when to disable, re-enable, or revoke access to your customer managed key at any time.

To check that the AWS managed key is symmetric:

$ aws kms describe-key --key-id <Key ID> --region <Region>

To check the customer managed key is symmetric:

$ aws kms describe-key --key-id <Key ID> --region <Region>

To enhance the data privacy and security posture, AWS recommends using data encryption as one of the access controls to your data. The following table (Table 1) shares the encryption support details for FSx for Lustre.

Linking FSx for Lustre cross-account Amazon S3 buckets using AWS Key Management Service (KMS)

To test the data protection using a customer managed key (CMK), one use case is integrating cross-account Amazon S3 bucket access from your FSx for Lustre file system where the S3 bucket has default encryption enabled using a customer managed CMK. For this use case, the FSx for Lustre file system is running in Account-A, the S3 bucket is in Account-B encrypted by the customer managed CMK, which is in Account-C.

To summarize, here are the steps to follow for the preceding use case:

Step 1: Allow FSx for Lustre file system in Account-A to access the Amazon S3 bucket in Account-B by editing the S3 bucket policy.

Step 2: Share the AWS KMS key across Account-A (Amazon FSx for Lustre) and Account-B (Amazon S3) by modifying the key policy.

Step 3: Enable Default Encryption in your S3 bucket to use the KMS key. Use the shared AWS KMS key to encrypt objects stored in S3 in Account-B.

Step 4: Upload a test file into an encrypted S3 bucket and try to read the data in it.

Step 5: Confirm if reading the data from the file will fail using an AWS CloudTrail event.

Step 6: Modify the AWS KMS key policy to grant permission to the FSx Service Link Role. Then create an AWS KMS grant to encrypt and decrypt the data and read the data from the encrypted file (S3 object).

Step 7: Test S3 exports using lfs_hsm commands.

Figure 1: Amazon FSx multi-account use case

Step 1: Allow permissions to Account-A using Amazon Resource Name (ARN) arn:aws:iam::Account-A:root or limited permission to allow the Amazon FSx service-linked role on the S3 bucket in Account-B via the S3 bucket policy.

Amazon FSx does not allow you to edit the AWSServiceRoleForAmazonFSx service-linked role, and after Amazon FSx creates a service-linked role, you cannot change the name of the role because other entities are referring to the role. For detailed information, see the description for editing a service-linked role for Amazon FSx.

If you are looking to limit access to the service-linked role instead of the account access, you first create a FSx for Lustre file system in Account-A that is linked to the S3 bucket in Account-B as a repository, which has principal:arn:aws:iam::account-A:root in its bucket policy. Once Amazon FSx creates the service-linked role, update the principal element in the bucket policy to restrict access to the Amazon FSx service-linked role. The service-linked role is visible in Account-A under IAM roles after the creation of the Amazon FSx file system. For example:

SLR ARN - "AWS":"arn:aws:iam::ACCOUNT-A:role/aws-service-role/s3.data-
source.lustre.fsx.amazonaws.com/AWSServiceRoleForFSxS3Access_fs-xxxxxxxxxx".

For Persistent 1, Scratch 1, and Scratch 2 deployment types, you get the service-linked role once you create the file system. For Persistent 2, the service-linked role is created only when you create the Data Repository Association. Once the service-linked role is created, you can update the principal in the S3 bucket policy with the Amazon FSx service-linked role to fine-tune access privileges.

Step 2: Share the AWS KMS key in Account-C to Account-A (FSx for Lustre) and Account-B (Amazon S3) by modifying the key policy.

Step 3: Enable the S3 bucket default encryption to use the KMS key. This allows you to encrypt the S3 bucket objects.

Step 4: Upload a test.txt file into the S3 bucket, which will get encrypted automatically as the default encryption is enabled on the S3 bucket using the AWS KMS key in Account-C. Make sure the AWS KMS key policy has permissions to allow the principal in Account-B to upload the file into the encrypted S3 bucket. Next, you need to mount the file system on an EC2 instance. Once the file system is mounted, you can check if your EC2 instance is able to load the data from the S3 bucket. Assuming the file system is mounted on /fsx, you can send list and read commands like the following:

To list the files in the mount:

$ ls /fsx/ 

Step 5: Read the files:

$ cat /fsx/test.txt

When there is no permission to decrypt the objects or files on the target S3 bucket, you will see the following error:

$ cat /fsx/test.txt
cat: error reading ‘/fsx’: No data available

In AWS CloudTrail, if you query for API call EventName: Decypt or EventSource: kms.amazonaws.com, it will show the following error message:

errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::Account-A:assumed-role/AWSServiceRoleForFSxS3Access_fs-xxxxxxxxxx/FSx is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:Account-C:key/2a1bae3d-a3e7-475b-acb2-b0dd92a5ff99 because no identity-based policy allows the kms:Decrypt action".

The previous error message indicates that the Amazon FSx service-linked role AWSServiceRoleForFSxS3Access_fs-xxxxxxxxxx does not have permissions to decrypt the S3 objects in Account-B using AWS KMS key Id: 2a1bae3d-a3e7-475b-acb2-b0dd92a5ff99 in Account-C.

Step 6: Ensure that the decryption is working for the FSx for Lustre file system. Modify the AWS KMS key policy to grant permissions to Account-A’s AWSServiceRoleForFSxS3Access_fs-xxxxxxxxxx service-linked role on the AWS KMS key and also grant permissions to an IAM identity in Account-A for performing AWS KMS grant API actions.

Be cautious about giving principal permissions to use your AWS KMS keys. Whenever possible, follow the least privilege principle.

 {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::Account-A:role/aws-service-role/s3.data-source.lustre.fsx.amazonaws.com/AWSServiceRoleForFSxS3Access_fs-xxxxxxxxxx"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:us-east-1:Account-C:key/a056d364-d490-473d-ad19-bd6a483d8096",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "s3.us-east-1.amazonaws.com",
                    "kms:CallerAccount": "Account-A"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::s3fsx/*"
                }
            }
        },
        {
            "Sid": "Allow IAM identity to perform KMS Grant operations",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::Account-A:user/Admin"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "arn:aws:kms:us-east-1:Account-C:key/a056d364-d490-473d-ad19-bd6a483d8096"
        }

Once the policy is applied, create an AWS KMS key grant for the Amazon FSx service-linked role in AWS Account-A. Due to the fact that the service-linked role cannot be modified, we are setting up a policy to allow the AWS principal to use AWS KMS keys in cryptographic operations. Grants are often used for temporary permissions because you can create one, use its permissions such as encrypt or, decrypt, and delete it without changing your AWS KMS key policy. AWS KMS grants are good until they are explicitly retired or revoked by the user. The following is an example command to create an AWS KMS grant:

$ aws kms create-grant --region us-east-1 \
       --key-id arn:aws:kms:us-east-1:Account-C:key/a056d364-d490-473d-ad19-bd6a483d8096 \
       --grantee-principal arn:aws:iam::Account-A:role/aws-service-role/s3.data-source.lustre.fsx.amazonaws.com/AWSServiceRoleForFSxS3Access_fs-xxxxxxxxx \
       --operations "Decrypt" "Encrypt" "GenerateDataKey" "GenerateDataKeyWithoutPlaintext" "CreateGrant" "DescribeKey" "ReEncryptFrom" "ReEncryptTo"

When creating an AWS KMS grant, it generates a token that is a unique, non-secret, variable-length, base64-encoded string that represents a grant. The above command will grant permissions to the grantee principal (FSx for Lustre service-linked role) to perform cryptographic operations such as decrypting the objects stored in an S3 bucket while importing into the file system, as well as encrypting the files while exporting it to S3 bucket using lfs hsm commands.

Be sure that the IAM entity in Account-A has permissions to perform the CreateGrant API action. If CreateGrant permissions are missing, then add the following statement to the IAM entity’s attached policy:

{
      "Sid": "AllowCreationOfGrantForTheKMS",
      "Effect": "Allow",
      "Action": "kms:CreateGrant",
      "Resource": "arn:aws:kms:us-east-1:Account-C:key/a056d364-d490-473d-ad19-bd6a483d8096"
}

Once the AWS KMS create-grant has been created, try to read the encrypted files (Objects) from the Lustre EC2 client.

$ cat test.txt
this is encrypted using cmk symmetric key

Step 7: Export encrypted files to the S3 bucket, on the Lustre EC2 client, and create a new file with some content using vi editor. Once the file is ready, use lfs hsm commands to export the file to the encrypted S3 bucket, then verify that object exported to the S3 bucket. Please refer to the following commands:

Create a new file with some content in it:

$ vi upload_encrypted_file.txt

Export the file to your data repository:

$ sudo lfs hsm_archive upload_encrypted_file.txt

Verify the file has successfully been exported:

$ sudo lfs hsm_action upload_encrypted_file.txt
upload_encrypted_file.txt: NOOP

Release data from your file system:

$ sudo lfs hsm_release upload_encrypted_file.txt

If there is no AWS KMS grant created or has been revoked, when you try to export the file or object to the encrypted S3 bucket, the command lfs hsm_archive will not return any error. However, in CloudTrail, you can see an AccessDenied error message for Event name: GenerateDataKey, and the object will not be exported and will not appear in the S3 bucket.

If you are automatically exporting updates to your S3 bucket from an Amazon FSx file system, you can see similar behavior when you create a file or object in the file system. If the grant is missing, the file will not be exported to the S3 bucket, and CloudTrail will show the same error. You can turn on logging CloudWatch logs to log errors when items fail to export correctly due to missing permissions.

You can integrate multiple encrypted Amazon S3 buckets with the same FSx for Lustre file system. Here, you need to update the bucket policy with the right S3 bucket ARNs and create-grant API with the right AWS KMS key ARNs.

Cleaning up

Once the test is completed, be sure to remove the deployed resources that facilitated the test. Delete the EC2 instance, S3, AWS KMS keys, and FSx for Lustre file system in the account to avoid incurring future costs on EC2 and Amazon FSx unless you are going to reuse the resources later.

To clean up your Amazon FSx for Lustre file system, you can run the CLI commands in the AWS CloudShell console. For Example:

$ aws fsx delete-file-system --file-system-id <<insert your file system id>>

Conclusion

In this blog, we showed you how to safeguard your data using FSx for Lustre’s encryption feature. This will help in linking FSx for Lustre cross-account Amazon S3 buckets using AWS Key Management Service (KMS) where the FSx for Lustre file system, S3 data repository, and KMS keys all are in different AWS accounts.

If your company or organization is mandating corporate or regulatory policies that require encryption of data and metadata at-rest, we recommend creating an encrypted file system and mounting your file system in EC2, which supports encryption of data in transit.

If you would like to learn more about FSx for Lustre, check out the Amazon FSx for Lustre product detail page and user guide.

Thanks for reading this blog post on FSx for Lustre file systems data protection. If you have any comments or questions, do not hesitate to leave them in the comments section.