Using Amazon S3 Object Lambda to Dynamically Watermark Images as They Are Retrieved

TUTORIAL

Overview

With Amazon S3 Object Lambda, you can add your own code to S3 GET, HEAD, and LIST requests to modify data as it is returned to an application. You can use custom code to modify the data returned by S3 GET requests to convert data formats (for example, XML to JSON), dynamically resize images, redact confidential data, and much more. You can also use S3 Object Lambda to modify the output of S3 LIST requests to create a custom view of objects in a bucket, and S3 HEAD requests to modify object metadata like object name and size.

The purpose of this tutorial is to show you how to get started with Amazon S3 Object Lambda. Many organizations store images in Amazon S3 that are accessed by different applications, each with unique data format requirements. In certain cases, images may need to be modified to include a watermark depending on the user accessing the image (for example, a paying subscriber can view images without watermarks, while a non-paying user receives a watermarked image).

In this tutorial, we will use S3 Object Lambda to add a watermark to an image as it is retrieved from Amazon S3. S3 Object Lambda can be used to modify data as it is retrieved from Amazon S3, without changing the existing object or maintaining multiple derivative copies of the data. By presenting multiple views of the same data and removing the need to store derivative copies, you can save on storage costs.

What you will accomplish

In this tutorial, you will:

  • Create an Amazon S3 bucket
  • Create an S3 Access Point
  • Create an AWS Lambda function to modify images
  • Create an S3 Object Lambda Access Point

Prerequisites

To complete this tutorial, you need an AWS account. Access this support page for more information on how to create and activate a new AWS account.

You can create an IAM user for the tutorial, or you can add permissions to an existing IAM user. To complete this tutorial, your IAM user must include the following permissions to access relevant AWS resources and perform specific actions:

  • s3:CreateBucket
  • s3:PutObject
  • s3:GetObject
  • s3:ListBucket
  • s3:CreateAccessPoint
  • s3:CreateAccessPointForObjectLambda
  • s3-object-lambda:WriteGetObjectResponse
  • lambda:CreateFunction
  • lambda:InvokeFunction
  • iam:AttachRolePolicy
  • iam:CreateRole
  • iam:PutRolePolicy

To clean up the resources you create in this tutorial, you will need the following IAM permissions:

  • s3:DeleteBucket
  • s3:DeleteAccessPoint
  • s3:DeleteAccessPointForObjectLambda
  • lambda:DeleteFunction
  • iam:DeleteRole

 

 AWS experience

Beginner

 Time to complete

20 minutes

 Cost to complete

Less than $1 (Amazon S3 pricing page)

 Requires

AWS account*

*Accounts that have been created within the last 24 hours might not yet have access to the resources required for this tutorial.

 Services used

Amazon S3, AWS Lambda, AWS IAM

 Last updated

February 1, 2023

Prerequisites

To complete this tutorial, you need an AWS account. Access this support page for more information on how to create and activate a new AWS account.

You can create an IAM user for the tutorial, or you can add permissions to an existing IAM user. To complete this tutorial, your IAM user must include the following permissions to access relevant AWS resources and perform specific actions: 

Implementation

Step 1: Create an Amazon S3 bucket

1.1 – Sign in to the Amazon S3 console

1.2 – Create an S3 bucket

  • Select Buckets from the Amazon S3 menu in the left navigation pane and then choose the Create bucket button.

1.3

  • In the Bucket name field, enter a descriptive, globally unique name for your bucket. Select which AWS Region you would like your bucket created in. We will create another resource later in this tutorial that must be in the same AWS Region.
  • You can leave the remaining options with the default selections. Navigate to the bottom of the page and choose Create bucket.

Step 2: Upload an object

Now that your bucket is created and configured, you are ready to upload an image.

2.1 – Upload an object

  • From the list of available buckets, select the bucket name of the bucket you just created.

2.2

  • Next, make sure the Objects tab is selected. Then, from within the Objects section, choose the Upload button.

2.3 – Add files

  • Choose the Add files button and then select the image you would like to upload from your file browser.
  • If you’d like, you can upload this sample image.

2.4 – Upload

  • Navigate down the page and choose the Upload button.

2.5

  • After your upload completes and is successful, choose the Close button.

Step 3: Create an S3 Access Point

Create an Amazon S3 Access Point that will be used to support the S3 Object Lambda Access Point, which we will create later in the tutorial.

3.1 – Create an S3 Access Point

  • Navigate to the S3 console and select the Access Points menu option in the left navigation pane. Then, choose the Create access point button.

3.2

  • In the Properties section, enter a desired Access point name, and choose the Bucket name that you entered in Step 1 by selecting the Browse S3 button. Next, set the Network origin as Internet.

3.3

  • Keep all other defaults as is. Navigate to the bottom of the page and choose the Create access point button.

3.4

  • The S3 Access Point will now appear in the list when you navigate to Access Points in the left navigation pane.

Step 4: Create the Lambda function

  • Next, create a Lambda function that will be invoked when S3 GET requests are made through an S3 Object Lambda Access Point.
  • We will use AWS CloudShell from the AWS Management Console to build and test S3 Object Lambda. You can use your own computer or an AWS Cloud9 instance to build the solution if you meet these requirements:
    - Latest version of the AWS Command Line Interface (CLI)
    - Credentials to create AWS Lambda functions/layers and IAM role
    - Python 3.9
    - zip utility
    - jq utility

4.1 – Start a CloudShell terminal

Select the CloudShell icon in the top right menu in the AWS Management Console.

If a CloudShell introduction window appears, feel free to read the content and choose Close.

A new browser tab will open with the CloudShell terminal (similar to the following screenshot):

4.2 – Prepare CloudShell to deploy the Lambda function

  • Run the following code in the CloudShell to prepare the environment and deploy the Lambda layer with the Pillow module. Copy and paste the following code into CloudShell to install the required dependencies and deploy the Lambda function.
# Install the required libraries to build new python
sudo yum install gcc openssl-devel bzip2-devel libffi-devel -y
# Install Pyenv
curl https://pyenv.run | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
source ~/.bash_profile

# Install Python version 3.9
pyenv install 3.9.13
pyenv global 3.9.13

# Build the pillow Lambda layer
mkdir python
cd python
pip install pillow -t .
cd ..
zip -r9 pillow.zip python/
aws lambda publish-layer-version \
    --layer-name Pillow \
    --description "Python Image Library" \
    --license-info "HPND" \
    --zip-file fileb://pillow.zip \
    --compatible-runtimes python3.9

Note: When copying and pasting code, CloudShell will open a warning window to confirm that you want to paste multiline code. Select Paste.

This step can take 10–15 minutes to complete.

4.3 – Build the Lambda function

  • Download a TrueType font that will be used by the Lambda function to add a watermark to an image. Copy and paste the following commands into CloudShell.
wget https://m.media-amazon.com/images/G/01/mobile-apps/dex/alexa/branding/Amazon_Typefaces_Complete_Font_Set_Mar2020.zip
  • Extract the TrueType font that will be used to write the watermarked text in the image.
unzip -oj Amazon_Typefaces_Complete_Font_Set_Mar2020.zip "Amazon_Typefaces_Complete_Font_Set_Mar2020/Ember/AmazonEmber_Rg.ttf"
  • Create the Lambda code that will be used to process the S3 Object Lambda requests.
cat << EOF > lambda.py
import boto3
import json
import os
import logging
from io import BytesIO
from PIL import Image, ImageDraw, ImageFont
from urllib import request
from urllib.parse import urlparse, parse_qs, unquote
from urllib.error import HTTPError
from typing import Optional

logger = logging.getLogger('S3-img-processing')
logger.addHandler(logging.StreamHandler())
logger.setLevel(getattr(logging, os.getenv('LOG_LEVEL', 'INFO')))
FILE_EXT = {
    'JPEG': ['.jpg', '.jpeg'],
    'PNG': ['.png'],
    'TIFF': ['.tif']
}
OPACITY = 64  # 0 = transparent and 255 = full solid


def get_img_encoding(file_ext: str) -> Optional[str]:
    result = None
    for key, value in FILE_EXT.items():
        if file_ext in value:
            result = key
            break
    return result


def add_watermark(img: Image, text: str) -> Image:
    font = ImageFont.truetype("AmazonEmber_Rg.ttf", 82)
    txt = Image.new('RGBA', img.size, (255, 255, 255, 0))
    if img.mode != 'RGBA':
        image = img.convert('RGBA')
    else:
        image = img

    d = ImageDraw.Draw(txt)
    # Positioning Text
    width, height = image.size
    text_width, text_height = d.textsize(text, font)
    x = width / 2 - text_width / 2
    y = height / 2 - text_height / 2
    # Applying Text
    d.text((x, y), text, fill=(255, 255, 255, OPACITY), font=font)
    # Combining Original Image with Text and Saving
    watermarked = Image.alpha_composite(image, txt)
    return watermarked


def handler(event, context) -> dict:
    logger.debug(json.dumps(event))
    object_context = event["getObjectContext"]
    # Get the presigned URL to fetch the requested original object
    # from S3
    s3_url = object_context["inputS3Url"]
    # Extract the route and request token from the input context
    request_route = object_context["outputRoute"]
    request_token = object_context["outputToken"]
    parsed_url = urlparse(event['userRequest']['url'])
    object_key = parsed_url.path
    logger.info(f'Object to retrieve: {object_key}')
    parsed_qs = parse_qs(parsed_url.query)
    for k, v in parsed_qs.items():
        parsed_qs[k][0] = unquote(v[0])

    filename = os.path.splitext(os.path.basename(object_key))
    # Get the original S3 object using the presigned URL
    req = request.Request(s3_url)
    try:
        response = request.urlopen(req)
    except HTTPError as e:
        logger.info(f'Error downloading the object. Error code: {e.code}')
        logger.exception(e.read())
        return {'status_code': e.code}

    if encoding := get_img_encoding(filename[1].lower()):
        logger.info(f'Compatible Image format found! Processing image: {"".join(filename)}')
        img = Image.open(response)
        logger.debug(f'Image format: {img.format}')
        logger.debug(f'Image mode: {img.mode}')
        logger.debug(f'Image Width: {img.width}')
        logger.debug(f'Image Height: {img.height}')

        img_result = add_watermark(img, parsed_qs.get('X-Amz-watermark', ['Watermark'])[0])
        img_bytes = BytesIO()

        if img.mode != 'RGBA':
            # Watermark added an Alpha channel that is not compatible with JPEG. We need to convert to RGB to save
            img_result = img_result.convert('RGB')
            img_result.save(img_bytes, format='JPEG')
        else:
            # Will use the original image format (PNG, GIF, TIFF, etc.)
            img_result.save(img_bytes, encoding)
        img_bytes.seek(0)
        transformed_object = img_bytes.read()

    else:
        logger.info(f'File format not compatible. Bypass file: {"".join(filename)}')
        transformed_object = response.read()

    # Write object back to S3 Object Lambda
    s3 = boto3.client('s3')
    # The WriteGetObjectResponse API sends the transformed data
    if os.getenv('AWS_EXECUTION_ENV'):
        s3.write_get_object_response(
            Body=transformed_object,
            RequestRoute=request_route,
            RequestToken=request_token)
    else:
        # Running in a local environment. Saving the file locally
        with open(f'myImage{filename[1]}', 'wb') as f:
            logger.debug(f'Writing file: myImage{filename[1]} to the local filesystem')
            f.write(transformed_object)

    # Exit the Lambda function: return the status code
    return {'status_code': 200}
EOF
  • Create the Lambda zip file that contains the Python code and TrueType font file.
zip -r9 lambda.zip lambda.py AmazonEmber_Rg.ttf
  • Create the IAM role that attaches to the Lambda function.
aws iam create-role --role-name ol-lambda-images --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
  • Attach a predefined IAM policy to the IAM role created previously. This policy contains the minimum permissions required to run the Lambda function.
aws iam attach-role-policy --role-name ol-lambda-images --policy-arn arn:aws:iam::aws:policy/service-role/AmazonS3ObjectLambdaExecutionRolePolicy

export OL_LAMBDA_ROLE=$(aws iam get-role --role-name ol-lambda-images | jq -r .Role.Arn)

export LAMBDA_LAYER=$(aws lambda list-layers --query 'Layers[?contains(LayerName, `Pillow`) == `true`].LatestMatchingVersion.LayerVersionArn' | jq -r .[])
  • Create and upload the Lambda function.
aws lambda create-function --function-name ol_image_processing \
	--zip-file fileb://lambda.zip --handler lambda.handler --runtime python3.9 \
	--role $OL_LAMBDA_ROLE \
	--layers $LAMBDA_LAYER \
	--memory-size 1024

Step 5: Create an S3 Object Lambda Access Point

Create an S3 Object Lambda Access Point that will be used to access the image(s) stored in your S3 bucket.

5.1 – Create the S3 Object Lambda Access Point

In the General section, for the Object Lambda Access Point name, enter ol-amazon-s3-images-guide.

Ensure that the AWS Region for the S3 Object Lambda Access Point matches the AWS Region you specified when creating the S3 bucket in Step 1.3.

For the Supporting Access Point, specify the Amazon Resource Name (ARN) of the S3 Access Point that you created in Step 3.2 using the Browse S3 button.

Navigate down to view Transformation configuration. In the S3 APIs list, select the GetObject option.

Under Lambda function, specify ol_image_processing.

Next, navigate to the bottom of the page and choose Create Object Lambda Access Point.

Step 6: Download images from the S3 Object Lambda Access Point

After creating the S3 Object Lambda Access Point, we will open the image to verify that a watermark is properly added during the request.

6.1 – Open the S3 Object Lambda Access Point

  • Return to the list of S3 Object Lambda Access Points by choosing Object Lambda Access Points in the S3 console left navigation pane and then select the S3 Object Lambda Access Point that you created in Step 5.1. In this example, we chose the S3 Object Lambda Access Point to be ol-amazon-s3-images-guide.

Select the image that you uploaded in Step 2.4 and then choose the Open button.

A new browser tab will open with your image and a watermark.
 
All compatible images downloaded from the S3 Object Lambda Access Point will now include the watermarked text.


6.2 – Downloading the transformed image from the AWS CLI

  • You can also download the image using the AWS CLI. To do this, you need the Amazon Resource Name (ARN) of the S3 Object Lambda Access Point. In the S3 console, navigate to the Object Lambda Access Points page, select the name of the S3 Object Lambda Access Point, select the Properties tab, and choose the copy icon below Amazon Resource Name (ARN).

6.3 – Run the AWS CLI command from the CloudShell

From the CloudShell browser tab, enter the following:

aws s3api get-object --bucket <paste the ARN copied above here> --key <image filename here> <filename to write here>

6.4 – Download the image to your local computer

From CloudShell, choose Actions in the top right corner and select Download file.

Enter the file name that you defined in Step 6.3 when downloading the image from the S3 Object Lambda Access Point and choose Download.

Now, you can open the image from your local computer.

Note: Image viewers can differ by computer and operating system. Check with your administrator if you are unsure which application to use to open the image.

Step 7: Clean up resources

In the following, you will clean up the resources you created in this tutorial. It is a best practice to delete resources that you are no longer using so you do not incur unintended charges.

7.1 – Delete the S3 Object Lambda Access Point

  • Navigate to the S3 console and choose Object Lambda Access Points in the left navigation pane.
  • On the Object Lambda Access Points page, choose the radio button to the left of the S3 Object Lambda Access Point that you created in Step 5.1.

Choose Delete.

Confirm that you want to delete your S3 Object Lambda Access Point by entering its name in the text field that appears, and then choose Delete.

7.2 – Delete the S3 Access Point

  • In the left navigation pane of the S3 console, choose Access Points.
  • Navigate to the S3 Access Point that you created in Step 3.1, and choose the radio button next to the name of the S3 Access Point.
  • Choose Delete.

Confirm that you want to delete your access point by entering its name in the text field that appears, and then choose Delete.

7.3 – Delete the test object

  • Navigate to the S3 console and select the Buckets menu option in the left navigation pane. First, you will need to delete the test object from your test bucket. Select the name of the bucket you have been working with for this tutorial.
  • Select the checkbox to the left of your test object name, then choose the Delete button.
  • On the Delete objects page, verify that you have selected the proper object to delete and enter delete into the Permanently delete objects confirmation box. Then, choose the Delete objects button to continue.
Next, you will be presented with a banner indicating whether the deletion has been successful.

7.4 – Delete the S3 bucket

  • Next, choose Buckets from the S3 console menu in the left navigation pane. Select the radio button to the left of the source bucket you created for this tutorial, and then choose the Delete button.

Review the warning message. If you desire to continue deletion of this bucket, enter the bucket name into the Delete bucket confirmation box, and choose Delete bucket.

7.5 – Delete the Lambda function

  • In the AWS Lambda console, choose Functions in the left navigation pane.
  • Select the checkbox to the left of the name of the function that you created in Step 4.3.
  • Choose Actions, and then choose Delete. In the Delete function dialog box, choose Delete.

Conclusion

Congratulations! You learned how to use Amazon S3 Object Lambda to dynamically add a watermark to an image as it is retrieved, delivering the processed image back to the requesting client. You can customize the Lambda function for your use case to modify the data returned by S3 GET, HEAD, and LIST requests, with common use cases including customizing watermarks using caller-specific details, masking sensitive data, filtering certain rows of data, augmenting data with information from other databases, converting data formats, and much more.

Was this page helpful?

 Feedback

Next steps

Amazon S3 Storage Lens product page

Learn more about how S3 Object Lambda works, its benefits, and a specific use case of redacting sensitive data from an object in Amazon S3.

Amazon S3 Storage documentation

Explore the Amazon S3 User Guide to learn more about transforming objects with Amazon S3 Object Lambda.

Amazon S3 Object Lambda FAQs

Review our FAQs about using Amazon S3 Object Lambda to modify and process data as it is returned to an application.