Modify images cached in Amazon CloudFront using Amazon S3 Object Lambda

Delivering optimized content is critical to a positive end user experience. As data access requirements evolve, the end user may need a transformed version of the original content. The transformations may include masking an image’s metadata, watermarking, or resizing an image before returning it to the user. The object should be stored in its original format, while the transformed version should be cached at the edge for quick access. In many cases, both the original and transformed versions of an object are stored. When a new access requirement is added, existing objects need to be transformed again and a new version saved. This leads to unnecessary storage and data processing costs.

Customers use a combination of Amazon CloudFront and Amazon Simple Storage Service (S3) to publicly serve static content for websites, applications, and more. With Amazon S3 Object Lambda, you can add code to modify the data returned by standard Amazon S3 GET requests to resize and watermark images, customize metadata, and much more. Amazon S3 Object Lambda eliminates the need to create and store derivative copies of your data or to run expensive proxies, all with no changes required to your applications.

In this post, I show you how to build a solution that lets you dynamically modify images cached in Amazon CloudFront via Amazon S3 Object Lambda using Lambda@Edge. After you’ve built the solution, a user uses a public URL to retrieve multiple transformed versions of an image stored in a private Amazon S3 bucket. This access pattern allows you to quickly adapt to changing business needs by serving a derived set of images without spending time copying and converting your existing image library.

Solution overview

The user accesses an object via a public URL, which routes the request to the nearest edge location. If the object is not in the edge cache, Amazon CloudFront sends an origin request to retrieve the object. We configure the origin to be the Amazon S3 Object Lambda Access Point, which allows us to retrieve the transformed object. However, Amazon S3 Object Lambda requires all access to be made by authenticated principals (that is, no anonymous access) and over HTTPS. Amazon CloudFront can redirect HTTP to HTTPS, but we need a method of adding authentication the request.

Lambda@Edge is an Amazon CloudFront feature that lets you run code at the edge in response to events generated by Amazon CloudFront. In our solution, we use a Lambda@Edge function to intercept the origin request and sign the request using the Lambda execution credentials. The updated, authenticated request is returned to Amazon CloudFront, which sends the request to the Amazon S3 Object Lambda Access Point.

Amazon S3 Object Lambda receives the authenticated request and returns the transformed object to Amazon CloudFront, where the object is returned to the user and cached at the edge location. Subsequent requests for the object at this edge location receive the cached version of the transformed object, thus avoiding extra Amazon S3 Object Lambda executions.

Solution configuration and walkthrough

There are two methods of deploying this solution: infrastructure-as-code (IaC) using an open-sourced CDK project, or by configuring services in the AWS Console.

If you’d like to deploy the solution using IaC, please refer to the installation section of the CDK project’s README file for instructions. This deploys the entire solution in minutes and allows you to run an end-to-end workflow.

In the next sections, I show you how to set up the solution by configuring AWS services via the Console:

1. Create an AWS Lambda function for S3 Object Lambda.
2. Create a supporting Amazon S3 Access Point.
3. Create an S3 Object Lambda Access Point.
4. Grant AWS Lambda Function additional privileges.
5. Create and publish an AWS Lambda function for Lambda@Edge.
6. Configure Lambda@Edge function permissions and trust relationships.
7. Configure the Amazon CloudFront distribution.

Prerequisites

You will need an S3 bucket to store source images. Configure this bucket to block public access with encryption enabled.

1. Create an AWS Lambda function for Amazon S3 Object Lambda

This AWS Lambda function is responsible for accessing an object in Amazon S3, transforming it, and returning the transformed object to Amazon S3 Object Lambda. The function is subject to standard AWS Lambda quotas, and has up to 60 seconds to send a complete response to the caller through the WriteGetObjectResponse request. The original Amazon S3 object is accessed via the presigned URL generated by Amazon S3 Object Lambda.

Navigate to the AWS Lambda console. Confirm that the Region displayed in the console is the same as the prerequisite S3 bucket Region. This is because data transfer between AWS Lambda and Amazon S3 in the same Region is free.

Select Create function:

1. Give your AWS Lambda a descriptive name.
2. Confirm the runtime is the latest supported Node.js version, and the architecture is x86_64.
3. Select the Create function button at the bottom of the page.

In the following example, I’ve named the Lambda function images-exif-transform.

The AWS Lambda function, including dependencies, is available to download as a .zip file. In the Code Source section of the AWS Lambda editor, select Upload from, then .zip file, and select the .zip file you downloaded. Select Save. The source code can be viewed here.

After uploading the AWS Lambda source code, navigate to the Configuration tab, select General configuration, and select Edit:

1. Increase Memory to 512 MB.
2. Increase Timeout to 1 min.
3. Select Save.

This function returns different objects to Amazon S3 Object Lambda:

• If the object is not a supported image type, return an error.
• Return a modified image with no metadata, including stripping EXIF.
• If a parameter named showExif is set to true, then return a JSON of the EXIF data.

2. Create a supporting Amazon S3 Access Point

Amazon S3 Object Lambda requires a supporting Amazon S3 Access Point. This allows you to create unique access control policies to easily control access to data in Amazon S3.

Navigate to the Access Points section of the Amazon S3 console. Select Create access point:

1. Give your access point a unique name.
2. Select the bucket where you have stored the original images.
3. Under Network origin select Internet.
4. Under Block Public Access settings for this Access Point, confirm that block all public access is selected.

In the following example, I’ve named my Access Point my-images-blog-access-point.

5. For the Access Point policy, we want to grant our AWS Lambda function access to all objects via the Access Point. Note the trailing /object/* on the resource. This allows the principal (that is, our AWS Lambda execution role) access to the Amazon S3 objects via this Access Point.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "<Lambda execution role ARN>"
			},
			"Action": "s3:GetObject",
			"Resource": "<Access point ARN>/object/*"
		}
	]
}

6. Select Create access point.

3. Create an S3 Object Lambda Access Point

An Amazon S3 Object Lambda Access Point is associated with exactly one standard Amazon S3 Access Point. A GET request through this Access Point invokes the associated AWS Lambda function and returns a transformed object.

Navigate to the Object Lambda Access Points section of the Amazon S3 console. Select Create S3 Object Lambda Access Point:

1. Give your Amazon S3 Object Lambda Access Point a unique name.
2. Under Supporting Access Point, select the Access Point created earlier.
3. Under AWS Lambda function, select the function created earlier.

In the following example, I’ve named my Access Point my-images-blog-s3-object-lambda-ap.

4. For the Object Lambda Access Point policy, we want to add a statement that grants our Lambda function access to all objects via the Amazon S3 Object Lambda Access Point. Note: You can find the Amazon S3 Object Lambda Access Point Amazon Resource Name (ARN) just above the policy editor.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "<Lambda execution role ARN>"
			},
			"Action": "s3-object-lambda:GetObject",
			"Resource": "<S3 Object Lambda access point ARN>"
		}
	]
}

5. Select Create Object Lambda Access Point.

4. Grant AWS Lambda Function additional privileges

We need to grant the AWS Lambda function permission to write its response to Amazon S3 Object Lambda. We do this by adding policies to the AWS Lambda execution role. This is an AWS Identity and Access Management (IAM) role that grants the function permission to access AWS services and resources.

Navigate to the AWS Lambda console and select the Lambda function you created earlier. In the Lambda code editor, select the Configuration tab, then select Permissions. The execution role is a link to the right.

Select this role name, which launches a new window to the IAM service.

1. Select Add permissions, then Create inline policy.
2. In the Create policy page that opens, select the JSON tab.
3. Paste the following JSON in the editor.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3-object-lambda:WriteGetObjectResponse",
            "Resource": "<S3 Object Lambda access point ARN>",
            "Effect": "Allow"
        }
    ]
}

4. Select Review policy.
5. Give the policy a descriptive name (for example, S3OL-WriteObjectResponse)
6. Select Create policy.

Note the execution role’s ARN, because we reference it later in this post.

5. Create and publish an AWS Lambda function for Lambda@Edge

This AWS Lambda function runs at the edge using Lambda@Edge and performs the following:

• Receives the Amazon CloudFront origin request. This is the request that Amazon CloudFront sends to Amazon S3 Object Lambda in order to retrieve an object.
• Signs the request using its execution credentials.
• Returns the updated, signed request to Amazon CloudFront.

Note: This AWS Lambda function must be deployed in the us-east-1 Region. After we configure our Amazon CloudFront distribution to use this function, it propagates to all edge locations.

Navigate to the AWS Lambda console. Confirm that the Region displayed is us-east-1 (N. Virginia). Select Create function.

1. Give your Lambda function a descriptive name, (for example, edge-origin-request-signer).
2. For Runtime, choose Python 3.9 and verify the architecture is x86_64.
3. Select the Create function button at the bottom of the page.

In the code editor, paste the source code into lambda_function.py. Then select Deploy.

Lambda@Edge points to a specific version of an AWS Lambda function. To publish a version, in the AWS Lambda function editor, select the Versions tab, select Publish new version, then select Publish.

After publishing the version, you are redirected to the AWS Lambda version page. Note the Function ARN, which is used later in our Amazon CloudFront configuration.

6. Configure Lambda@Edge function permissions and trust relationships

The previous function’s execution role is used to sign anonymous requests from Amazon CloudFront. We grant policies to this role that allow it to:

• Get objects from the Amazon S3 bucket and S3 supporting Access Point (s3:GetObject).
• Get objects from the Amazon S3 Object Lambda Access Point (s3-object-lambda:GetObject).
• Invoke the AWS Lambda function associated with the Amazon S3 Object Lambda (lambda:InvokeFunction).
• Create log groups, log streams, and put log events to all regions as Lambda@Edge functions run in edge locations – that is, not just the Region where the function was created – (logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents).

Update the policy for this function using the role link from the AWS Lambda editor.

Select the JSON tab and replace the policy with the following.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:<account ID>:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:<account ID>:log-group:/aws/lambda/*<Edge function name>:*"
            ]
        },
        {
            "Action": "lambda:InvokeFunction",
            "Resource": "<Lambda ARN associated with S3 Object Lambda>:$LATEST",
            "Effect": "Allow"
        },
        {
            "Action": "s3:GetObject",
            "Resource": [
			"<S3 bucket ARN>/*",
			"<S3 supporting access point ARN>/object/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "s3-object-lambda:GetObject",
            "Resource": "<S3 Object Lambda access point ARN>",
            "Effect": "Allow"
        }
    ]
}

Select Review policy. It should look similar to the following:

Select Save changes.

Typically, an AWS Lambda function role is assumed by the AWS Lambda service. In our use case, the Lambda@Edge service also assumes this role.

From the IAM window, select the Trust relationships tab, and select Edit trust policy. Replace the service section of the JSON with the following:

"Service": [
    "lambda.amazonaws.com",
    "edgelambda.amazonaws.com"
]

Select Update policy.

7. Configure the Amazon CloudFront distribution

Amazon CloudFront is a low-latency content delivery network (CDN) that delivers data through 410+ globally dispersed Points of Presence (PoPs) with automated network mapping and intelligent routing. By default, static content is cached for 24 hours at edge locations. In our use case, Amazon S3 Object Lambda processes each requested image once per 24 hours (twice if we request the JSON EXIF data).

Create a cache policy

An Amazon CloudFront cache policy determines whether an HTTP request results in a cache hit (that is, the object is served to the viewer from the Amazon CloudFront cache) via the cache key. Since our Amazon S3 Object Lambda function supports the showExif query string, we want to use that as part of the cache key.

Navigate to the Policies section of the Amazon CloudFront console. Select the Cache tab, then select Create cache policy.

1. Give the cache policy a descriptive name (i.e., object-lambda-query-string).

2. For Query strings, select Include specified query strings, and add showExif to the Allow list.

3. Select Create.

Create the Amazon CloudFront distribution

Navigate to the Amazon CloudFront console. Select Create distribution.

Origin:

1. For Origin domain, type in the URL prefix for our Amazon S3 Object Lambda Access Point. The format is:

<s3 object lambda access point name>-<account ID>.s3-object-lambda.<access point region>.amazonaws.com

• • For example:

my-images-blog-s3-object-lambda-ap-123456789012.s3-object-lambda.us-west-2.amazonaws.com

2. For Protocol, select HTTPS only.
3. For Minimum origin SSL protocol, select 2.

Default cache behavior:

1. For Viewer protocol policy, select Redirect HTTP to HTTPS.
2. Under Cache key and origin requests, for cache policy, select the policy created in the earlier section.

Function associations:

1. Next to origin request, select Lambda@Edge.
2. In the Function ARN/Name text field, insert the ARN of the Lambda@Edge Lambda version we created earlier. Note: Make sure the ARN ends with :#, where # is the version number.
3. Select Create distribution.

The distribution is ready when the distribution state changes from Deploying to a timestamp.

Solution testing

Upload an image containing EXIF data to the S3 image storage bucket. Typically, a picture taken by a mobile phone has this data. A sample picture is provided here.

Navigate to the Amazon CloudFront domain name and append the picture name and path to access the image anonymously through Amazon S3 Object Lambda.

In the following screenshot, the original picture is on the left, while the image downloaded through Amazon CloudFront is on the right. Notice that the EXIF data has been stripped.

Add a search parameter &showExif=true to the end of the URL in order to return a JSON of the image EXIF data.

Cleaning up

To delete the resources created by the automated CDK deployment, refer to the uninstall section of the repository. To delete the resources created using the walkthrough, clean up the following:

• Amazon CloudFront: Delete the distribution.
• AWS Lambda functions: Refer to the clean-up section of the AWS Lambda guide.
• Amazon S3 bucket: Delete the bucket.

Conclusion

In this post, I showed you how to use a combination of Amazon S3 Object Lambda and Amazon CloudFront Lambda@Edge to serve transformed versions of images stored in Amazon S3. This post described the general building blocks of anonymous Amazon S3 Object Lambda access, and can be extended easily using other transformation functions. With Amazon CloudFront, you can extend functionality by making use of HTTP headers or search parameters, like we have done with the showExif=true parameter.

With this solution, you can store original images in Amazon S3 without having to worry about future data-access requirements or running backfills for transformations. You also save on storage costs by storing fewer derivative copies of data. By adapting this solution to your use case, you can deliver optimized content to end users securely and consistently without the heavy lifting of maintaining an object transformation pipeline.

To learn more about Amazon S3 Object Lambda, visit the S3 User Guide. To learn more about Amazon CloudFront Lambda@Edge, visit the CloudFront user guide.