Encrypting objects with Amazon S3 Batch Operations

bucket string,
key string,
version_id string,
is_latest boolean,
is_delete_marker boolean,
size bigint,
last_modified_date timestamp,
e_tag string,
storage_class string,
is_multipart_uploaded boolean,
replication_status string,
encryption_status string,
object_lock_retain_until_date timestamp,
object_lock_mode string,
object_lock_legal_hold_status string

bucket string,
key string,
version_id string,
is_latest boolean,
is_delete_marker boolean,
encryption_status string

5. Choose Add, Next.

6. Choose Create Table.

7. After creating your table, select the database that you specified earlier and run the query on the newly created table. Run the following query to get the buckets, key, and version IDs for only the unencrypted objects:

SELECT distinct bucket, key, version_id FROM rwx_tabledemo
where encryption_status != 'SSE'

If you have more than one S3 Inventory report saved in this location, run the query with Select Distinct to only select unique records and eliminate duplicates.

8. Download the results, remove the header row, and save as a CSV in your S3 bucket.

Setting up and running your S3 Batch Operations job

Now that you have your filtered CSV lists of S3 objects, you can begin the S3 Batch Operations job to encrypt the objects.

A job refers collectively to the list (manifest) of objects provided, the operation performed, and the specified parameters. The easiest way to encrypt this set of objects is by using the put copy operation and specifying the same destination prefix as the objects listed in the manifest. This either overwrites the existing objects in an unversioned bucket or creates a newer version of the objects, depending on the bucket’s versioning status.

As part of copying the objects, specify that S3 should encrypt the object with SSE-S3 or SSE-KMS encryption. This job copies the objects, so all your objects show an updated creation date upon completion, regardless of when you originally added them to S3. You also must specify the other properties for your set of objects as part of the S3 Batch Operations job, including object tags and storage class.

To create your job and encrypt your objects:

1. In the S3 console, choose Buckets, Batch Operations, Create Job.

2. Choose the Region where your objects are stored and the CSV file created earlier from the S3 Select or Athena results. If your manifest contains version IDs, check that box. Choose Next.

3. Choose PUT copy, the bucket containing the objects listed in your manifest, the desired encryption type (such as SSE-S3), storage class, and the other parameters as desired. The parameters specified at this step will apply to all operations performed on the objects listed in the manifest. Choose Next.

4. Give your job a description, set its priority level, choose a report type, and specify its IAM role. For more information, see Managing Batch Operations Jobs.

1. 1. In the IAM console, choose Role, Create Role.
   2. Choose AWS service, S3, S3 Batch Operations. Choose Next: Permissions.
   3. Choose Create Policy, JSON and paste in the permissions from the PUT copy object section from Granting Permissions for Amazon S3 Batch Operations. Modify the buckets to match the S3 location of your manifest, objects, and report
   4. Choose Review policy.
   5. Use the following example policy:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "CopyObjectsToEncrypt",
            "Effect": "Allow",
            "Action": [
              "s3:PutObject",
              "s3:PutObjectTagging",
              "s3:PutObjectAcl",
              "s3:PutObjectVersionTagging",
              "s3:PutObjectVersionAcl",
              "s3:GetObject",
              "s3:GetObjectAcl",
              "s3:GetObjectTagging",
              "s3:GetObjectVersion",
              "s3:GetObjectVersionAcl",
              "s3:GetObjectVersionTagging"
            ],
            "Resource": "arn:aws:s3:::{source_and_destination_bucket for copy}/*"
          },
          {
            "Sid": "ReadManifest",
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::{manifest_key}"
          },
          {
            "Sid": "WriteReport",
            "Effect": "Allow",
            "Action": [
              "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::{reportbucket}/*"
          }
        ]
      }

   6. Add the remaining parameters and choose Create Policy.
   7. With your S3 Batch Operations policy now complete, return to the Role tab and Create Role wizard to attach the newly created policy to the IAM role in the Attach Permissions Policies section of the wizard.

5. Return to your S3 Batch Operations create job workflow, refresh the list of IAM roles, select your newly created role, and choose Next.

6. Check that everything is correct before choosing Create Job.

The setup wizard automatically returns you to the S3 Batch Operations section of the S3 console. Your new job transitions from the New state to the Preparing state as S3 begins the process. During the Preparing state, S3 reads the job’s manifest, checks it for errors, and calculates the number of objects. Depending on the size of the manifest, reading can take minutes or hours—roughly two minutes per one-million objects.

After S3 finishes reading the job’s manifest, the job moves to the Awaiting your confirmation state. Check the number of objects in the manifest and choose Confirm the job.

After the job begins running, you can check its progress through the console dashboard view or by selecting the specific job. You can also choose to perform this or any other step through the AWS CLI, SDKs, or APIs.

You are charged for S3 Batch Operations jobs, objects, and requests in addition to any charges associated with the operation that S3 Batch Operations performs on your behalf, including data transfer, requests, and other charges. Writing new objects to the bucket, as is done here while copying, could also initiate S3 Replication, event-driven workflows, or other actions. You can find more information on S3 pricing on the S3 pricing page.

When the job completes, view the Successful and Failed object counts to confirm that everything performed as expected. For the exact cause of any Failed objects, see your job report.

To recap, you created a list of objects in your bucket, filtered that list to include only unencrypted objects, and used S3 Batch Operations to encrypt those objects using the copy operation. If you use a versioned bucket, the S3 Batch Operations job performed previously creates new encrypted versions of your objects. It also maintains the previous unencrypted versions. To delete the old versions, you should set up an S3 lifecycle expiration policy for noncurrent versions as described in Lifecycle Configuration Elements.

Other operations

By the way, similar steps can be used to do other things at large scale including adding object tags to objects for lifecycle transitions or restoring objects from Amazon S3 Glacier. In addition to encrypting files described previously, you can modify the process to do the following tasks:

• Adding object tags to drive lifecycle actions: Instead of writing a lifecycle rule for an entire bucket or prefix, tags allow you to perform actions on a targeted set of objects. You can filter objects by name, size, or any combination of these and other properties to identify objects for transition or expiration actions. You could, for example, tag all objects over 3 MB and only lifecycle those tagged objects to S3 Glacier to save storage costs.

• Restoring objects from S3 Glacier: Similarly, filtering your objects helps identify only those objects that you must restore from S3 Glacier. This filtering can be done based on the suffix of the object (such as, .jpg, .csv) or by other properties.

Things to know

S3 Batch Operations performs the same operation across all the objects listed in the manifest. If you want your copied objects to have different storage classes or other properties, create multiple jobs. Or, use an AWS Lambda function and have it assign specific properties to each object.

S3 Batch Operations is available today in all commercial AWS Regions except Asia Pacific (Osaka). S3 Batch Operations is also available in both of the AWS GovCloud (US) Regions and the AWS China Regions.

Using the copy operation

The copy operation creates new objects with new creation dates, which can affect lifecycle actions like archiving. If you copy all objects in your bucket, all the new copies have identical or similar creation dates. To further identify these objects and create different lifecycle rules for various data subsets, consider using object tags.

Failure threshold

If greater than 50% of a job’s object operations fail after more than 1,000 operation attempts, the job automatically fails. This provides quicker feedback and reduces the cost of failed requests. Check your final report to identify the cause of the failures.

Use S3 Select to save money

S3 Select is an S3 capability designed to pull out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that must access data in S3.

Most applications must retrieve the entire object and then filter out only the required data for further analysis. S3 Select enables applications to offload the heavy lifting of filtering and accessing data inside objects to the S3 service. By reducing the volume of data loaded and processed by your applications, S3 Select can improve the performance of most applications that frequently access data from S3.

Conclusion

In this post, I demonstrated how fast and easy it is to use S3 Batch Operations to encrypt existing data in your S3 buckets, even sorting existing objects to filter out data you have already encrypted. This process can save you time and money while allowing you to complete operations like encrypting all existing objects.

Thanks for reading this post and getting started with S3 Batch Operations. I can’t wait to hear your feedback and feature requests for S3 Batch Operations in the comments.