New Event Notifications for Amazon S3

Many AWS customers have been building applications that use Amazon Simple Storage Service (Amazon S3) for cost-efficient and highly scalable persistent or temporary object storage. Some of them want to initiate processing on the objects as they arrive; others want to capture information about the objects and log it for tracking or security purposes. These customers have been asking for a reliable and scalable way to be notified when an S3 object is created or overwritten.

S3 Event Notifications
Today we are launching a new event notification feature for S3. The bucket owner (or others, as permitted by an IAM policy) can now arrange for notifications to be issued to Amazon Simple Queue Service (Amazon SQS) or Amazon Simple Notification Service (Amazon SNS) when a new object is added to the bucket or an existing object is overwritten. Notifications can also be delivered to AWS Lambda for processing by a Lambda function. Here’s the general flow:

Here’s what you need to do in order to start using this new feature with your application:

1. Create the queue, topic, or Lambda function (which I’ll call the target for brevity) if necessary.
2. Grant S3 permission to publish to the target or invoke the Lambda function. For SNS or SQS, you do this by applying an appropriate policy to the topic or the queue. For Lambda, you must create and supply an IAM role, then associate it with the Lambda function.
3. Arrange for your application to be invoked in response to activity on the target. As you will see in a moment, you have several options here.
4. Set the bucket’s Notification Configuration to point to the target.

From that point forward the events will be reliably delivered to the target as appropriate.

Notifications are configured at the bucket level and apply to all of the objects in the bucket (we plan to provide control at a finer level at some point). You can elect to receive notification for any or all of the following events:

• s3:ObjectCreated:Put – An object was created by an HTTP PUT operation.
• s3:ObjectCreated:Post – An object was created by HTTP POST operation.
• s3:ObjectCreated:Copy – An object was created an S3 copy operation.
• s3:ObjectCreated:CompleteMultipartUpload – An object was created by the completion of a S3 multi-part upload.
• s3:ObjectCreated:* – An object was created by one of the event types listed above or by a similar object creation event added in the future.
• s3:ReducedRedundancyObjectLost – An S3 object stored with Reduced Redundancy has been lost.

Notification Details
Each notification is delivered as a JSON object with the following fields:

• Region
• Timestamp
• Event Type (as listed above)
• Request Actor Principal ID
• Source IP of the request
• Request ID
• Host ID
• Notification Configuration Destination ID
• Bucket Name
• Bucket ARN
• Bucket Owner Principal ID
• Object Key
• Object Size
• Object ETag
• Object Version ID (if versioning is enabled on the bucket)

For use cases that require strong consistency on S3, it is a good practice to use versioning when you are overwriting objects. With versioning enabled for a bucket, the event notification will include the version ID. Your event handle can use the ID to fetch the latest version of the object. The notification also includes the ETag of the new object. Your code can Get the object and verify the ETag before processing. If the ETags do not match, you can defer processing by posting the message back to the SNS or SQS target. Note that eventual consistency is a concern only if your application allows existing objects to be overwritten.

Configuring Notifications Using the Console
Here is how to configure an event notification using the AWS Management Console. I have a bucket named jbarr-upload and I want to send notifications to an SNS topic named jbarr-upload-notify. I have already configured the topic to send an email to me (this would generate an overwhelming amount of email and would not be suitable for an actual application, but it makes for a good demo). I start by granting permission for S3 to publish to the topic:

Then I configure the bucket to send notification to my topic:

I can use the menu to choose the event types that are of interest to me:

For testing purposes, I use the console to upload an object:

Here’s the resulting email notification (I’ve formatted the JSON for readability in order to get my point across):

As I noted earlier, email notification is inadvisable for production-scale applications.

I’m confident that you can pick up from where I left off and start integrating this feature into your own applications. You can, of course, use the AWS SDKs to configure and manage notifications.

Things to Know
Here are a couple of things to keep in mind as you start to think about the best way to use these new notifications as part of your application:

• Delivery Latency – Notifications are delivered to the target in well under a second.
• Cost – There is no charge for this feature. You will pay the usual messaging and execution charges for SQS, SNS, and Lambda (many applications can run within the AWS Free Tier).
• Regions – The bucket and the target (Amazon SQS, Amazon SNS, or Lambda) must reside in the same AWS Region.
• Event Types – You can configure one notification per event type per bucket.
• Delivery Reliability – S3 is designed to deliver notifications with a very high degree of reliability. It includes built-in backoff and retry mechanisms to deal with momentary issues that might affect the deliverability of messages to any of the three types of targets.
• Additional Event Types – We expect to add additional event types over time and your feedback will help us to prioritize our work. Please feel free to tell us more about your needs in the S3 Forum.

Availability
This feature is available now and you can start using it today! I am looking forward to hearing all about the interesting use cases that you come up with.

— Jeff;