AWS Compute Blog

Messaging Fanout Pattern for Serverless Architectures Using Amazon SNS

Sam Dengler, Amazon Web Services Solutions Architect

Serverless architectures allow solution builders to focus on solving challenges particular to their business, without assuming the overhead of managing infrastructure in AWS. AWS Lambda is a service that lets you run code without provisioning or managing servers.

When using Lambda in a serverless architecture, the goal should be to design tightly focused functions that do one thing and do it well. When these functions are composed to accomplish larger goals in microservice architectures, the complexity shifts from the internal components to the external communication between components. It’s all too easy to accidentally back into an architecture that is rigid to change because components are too knowledgeable of each other via the communication paths between them.

Solution builders can address this architectural challenge by using messaging patterns, resulting in loosely coupled communication between highly cohesive components to manage complexity in serverless architectures. As introduced in the recent Building Scalable Applications and Microservices: Adding Messaging to Your Toolbox post, a common approach when one component wishes to deliver the same message to multiple receivers is to use the fanout publish/subscribe messaging pattern.

The fanout pattern for message communication can be implemented in code. However, depending on your requirements, alternative solutions exist to offload this undifferentiated responsibility from the application. Amazon SNS is a fully managed pub/sub messaging service that lets you fan out messages to large numbers of recipients.

In this post, I review a serverless architecture from PlayOn! Sports as a case study for migration of fanout functionality from application code to SNS.

PlayOn! Sports serverless video processing platform

PlayOn! Sports is one of the nation’s leading high school sports media companies. They operate a comprehensive technology platform, enabling high-quality, low-cost productions of live sports events for the NFHS High School Sports Network.

At the 2014 AWS re:Invent conference, Lambda was announced. The PlayOn! Sports technology team recognized the parallels between serverless demos featuring image processing using ImageMagick to video processing using ffmpeg.

At the time, PlayOn! Sports was broadcasting live video with adaptive bit rates, requiring a transcoding of the video stream to multiple quality levels for consumption on desktop, mobile, and connected devices. This is not unusual for an internet media company. However, with over 50,000 live broadcasts produced in 2014, the traditional media and entertainment technological approaches and pricing models would not work.

After some consultation with the Lambda team to validate support for custom binary execution, PlayOn! Sports moved forward with the development of a new serverless video processing platform according to the architecture diagram below.

In the architecture, a laptop in the field captures the video from a camera source and divides it into small fragments, according to the HLS protocol. The fragments are published to Amazon S3, through an Amazon CloudFront distribution for accelerated upload. When the file has been written to S3, it triggers a Lambda function to initiate the video segment processing.

Video transcoding fanout implementation in Lambda

Given Lambda’s integration growth across AWS, it’s easy to forget that it did not include managed integration with SNS when it was announced in November 2014.

PlayOn! Sports was actively experimenting with approaches to video processing to address quality control, audience growth, and cost constraints. Lambda was a great tool for rapid innovation. A goal of the architecture design was the ability to add and remove video processing alternatives to the workflow, using the fanout pattern to identify optimal solutions. Below is example code from the initial implementation:

import json
import logging
import boto3

logger = logging.getLogger('boto3')
logger.setLevel(logging.INFO)

client = boto3.client('lambda')
fanout_functions = ['media_info', 'transcode_audio']

def lambda_handler(event, context):
    logger.info(json.dumps(event))
    logger.info('fanout_functions: %s', fanout_functions)

    for fanout_function in fanout_functions:
        logger.info('invoke: %s', fanout_function)
        response = client.invoke(
            FunctionName=fanout_function,
            InvocationType='Event',
            Payload=json.dumps(event)
        )
        logger.info('response: %s', response)

    return 'done'

Each Lambda function is invoked asynchronously, injecting the same S3 event that triggered the original Lambda function. For example, the media_info Lambda function could be scaffolded similar to the following code snippet:

import json
import logging
import boto3

logger = logging.getLogger('boto3')
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info(json.dumps(event))

    # MediaInfo Processing
    # see: https://aws.amazon.com/blogs/compute/extracting-video-metadata-using-lambda-and-mediainfo/

    return 'done'

Refactoring fanout implementation using SNS

The PlayOn! Sports development team was familiar with SNS, but had not used it previously to support system-to-system messaging patterns. After the announcement of SNS triggering of Lambda functions, the PlayOn! Sports team planned to migrate to the new feature to offload the overhead of managing the fanout Lambda function.

When invoking a Lambda function, SNS wraps the original event with SNSEvent. The Lambda function can be refactored by adding a function to parse the S3 event from SNSEvent, as seen in the following code:

import json
import logging
import boto3

logger = logging.getLogger('boto3')
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    s3_event = parse_event(event)
    logger.info(json.dumps(s3_event))

    # MediaInfo Processing
    # see: https://aws.amazon.com/blogs/compute/extracting-video-metadata-using-lambda-and-mediainfo/

    return 'done'


def parse_event(event):
    record = event['Records'][0]
    if 'EventSource' in record and record['EventSource'] == 'aws:sns':
        return json.loads(record['Sns']['Message'])

    return event

This Lambda function modification can be authored, tested, and deployed before enabling the SNS integration to verify that the existing Lambda fanout execution path continues to operate as before. The Lambda function invocation can now be transferred from the fanout Lambda function to SNS without disruption to S3 processing.

As the diagram below shows, the resulting architecture is similar to the original. The exception is that objects written to S3 now trigger a message to be published to an SNS topic. This sends the S3 event to multiple Lambda functions to be processed independently.

Sample architecture deployment using AWS CloudFormation

AWS CloudFormation gives developers and system administrators an easy way to create and manage a collection of related AWS resources. CloudFormation provisions and updates resources in an orderly and predictable fashion. To launch the CloudFormation stack for the sample fanout architecture, choose the following button:

 

 

Follow these steps to complete the architecture deployment:

  1. If necessary, sign into the console for your account when prompted.
  2. On the Select Template page, choose Next.
  3. Under Parameters, for S3BucketName, enter a globally unique name.
  4. For SnsTopicName, enter a region-unique name.
  5. Choose Next, Next.
  6. Select the checkbox for I acknowledge that AWS CloudFormation might create IAM resources, and choose Create.

After the stack has completed creation, you can test both paths of execution by uploading files to the S3 bucket that you created. Uploading a file to the “/uploads/lambda/” directory in S3 triggers the Lambda fanout function. Uploading a file to the “/uploads/sns/” directory in S3 triggers the SNS fanout execution path. You can verify execution by monitoring the Lambda function outputs in CloudWatch Logs.

Conclusion

In this post, I reviewed the fanout messaging pattern and options for its inclusion in a serverless architectures using Lambda application code and SNS. Using the PlayOn! Sports serverless video processing pipeline use case, I demonstrated how easy it is to refactor an existing application to use the SNS fanout approach.

I also provided a sample architecture in CloudFormation that you can run in your own account. Try it out and expand the sample architecture by adding other Lambda functions to the SNS topic, to demonstrate the flexibility of the fanout messaging pattern!

You can get started with SNS using the AWS Management Console, or the SDK of your choice. For more information about how to use SNS fanout messaging, see the following resources:

If you have questions or suggestions, please comment below.