Interactive advertising with Amazon IVS

The ability to monetize live streaming content can be a challenge in the media and entertainment industry. For example, when streaming live content via social media platforms, options may not be available to advertise precisely the way you want within your show or event.

In this post, the authors explain and demonstrate how to recognize objects in a live video stream, and to display a banner with a QR code to the viewer that allows them to learn more about or purchase the recognized product.

Solution overview

With Amazon Interactive Video Service (Amazon IVS), it is possible to create a live streaming experience with very few steps from the AWS console or from the command line. Using a video player such as Video.js provides low latency (between 2 and 5 seconds) and allows for real-time interactivity with an audience.

This example uses Amazon Rekognition Customer Labels to create two sample product labels. Begin the live stream (using software such as OBS) to Amazon IVS, which we configure to write a thumbnail image to Amazon S3 once every 5 seconds. Each picture on Amazon S3 is used to trigger an AWS Lambda function to analyze the image using an Amazon Rekognition custom model. If one of the two product logos is identified in the image, the AWS Lambda function will use the Amazon IVS put_metadata API to display the information relating to the product: Qr-code, description, image, and other information. At this point the same function will invoke an AWS Step Function which checks the clearing of the message after two minutes (configurable) and the impossibility of adding other messages for ten minutes (configurable) to avoid an annoying intermittent effect for the viewer.

Steps

You can follow the detailed guide in the GitHub repository in order to create:

• Amazon Rekognition Custom Label
• Amazon IVS channel
• AWS Lambda functions
• Amazon DynamoDB table
• AWS Step Functions workflow
• Static components

The creation of the Amazon Rekognition Custom Labels model lets you train the service to recognize objects that will trigger an advertising overlay to display within a live stream. You can include recognition of up to 50 different objects in your model. In the GitHub repository, you can find the sample images used to train the model in this blog post and a step-by-step instructional video for its creation.

IMPORTANT: You need to start the Amazon Rekognition Custom Labels model before beginning custom object detection, and make sure to stop it after streaming to avoid incurring additional cost.

When your Amazon IVS stream starts, images are saved every five seconds in the Amazon S3 bucket, which acts as a trigger to call the Lambda function and will use Amazon Rekognition’s DetectCustomLabels API to determine if any of the objects in the Custom Labels model are present in the image.

When an object is recognized, the AWS Lambda function instructs the video player to display an overlay formatted according to the CSS provided in the repository, and display any specified QR code. You can customize the CSS to meet your business needs.

The Lambda function also starts a Step Functions workflow to clear the overlay after a configurable amount of time, and to make sure that overlays do not trigger too close together.

For example, once a product is detected, the timed metadata with the QR code and other product information are inserted in the stream, which triggers display of the overlay. After two minutes, the overlay is removed, and in the following 10 minutes, even if a new object is detected, no additional overlays are displayed.

This allows you not only to control the recognition mode, but also to offer a non-invasive experience to your users.

The result

These two short videos provide a demonstration of the solution.

First object detected: You can find an example in the following link where the solution recognizes the twitch logo and inserts metadata where you can find the qr-code, information, description, etc.

Twitch Video demonstration

Second object detected: In the following pictures and in the link , you can see the detection of AWS logo.

AWS Video demonstration

Cleaning up

When you’re finished testing the deployment, it’s important to delete the created resources:

• Lambda functions: ivs-clear-text, Ivs-clear-ttxexpired-object1, Ivs-clear-ttxexpired-object1, ivs-lambda-process
  Amazon Step Functions: ivs-clear-step-object1 and ivs-clear-step-object2
• DynamoDB table: ivs-objectstatus
• The Amazon IVS channel
• The Amazon Rekognition Custom Label model

Conclusion

In this post, we described how to create a model that recognizes products or brands in a live video stream in real time and can suggest an advertising message to viewers or a link to a more in-depth site through the use of QR codes.

This solution is adaptable to monetize video content, customizing the quantity and frequency of advertising views.

To learn more about the resources used in this solution, visit the documentation for Amazon Rekognition, AWS Lambda , AWS Step Function, Amazon IVS , Amazon S3 and Amazon DynamoDB. Also, visit the GitHub repository for additional details on how to deploy this demo.