Posted On: Aug 13, 2020
(Note: This post was updated February 23, 2021 for clarity on functionality)
AWS Lambda now supports Amazon Managed Streaming for Apache Kafka (Amazon MSK) as an event source, giving customers more choices to build serverless applications with streaming data. Customers can build Apache Kafka consumer applications with Lambda functions without needing to worry about infrastructure management. Amazon MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.
Lambda makes it easy to process data streams from Amazon Kinesis Data Streams or Amazon DynamoDB Streams. Now, customers can also build applications triggered from Apache Kafka topics using Amazon MSK. Messages are processed in batches of up to 10,000 records, which may contain messages from one or more partitions, up to a maximum payload size of 6MB. Messages within a partition are always processed in order.
To get started, select the Amazon MSK topic as the event source for your Lambda function through the AWS Management Console, AWS CLI, AWS SAM, or AWS SDK for Lambda. Amazon MSK as a Lambda event source is available in all AWS Regions where both AWS Lambda and Amazon MSK are available, with the exception of the AWS China Regions and the AWS GovCloud (US) Regions. This feature requires no additional charge to use. You pay for the Lambda invocations triggered by an Apache Kafka topic. To learn more about building an Apache Kafka consumer application with Lambda, read the Lambda Developer Guide.