Using Amazon CloudWatch with Amazon EventBridge for cross-account event monitoring

We often talk about event driven architectures where an event is something that happens within your application or architecture. It could be a new file received by your application or when there is an alert triggered by high CPU utilization. We can act on these events by scanning the file contents or scaling out more Amazon Elastic Compute Cloud instances to provide more resources. These actions are driven by events within the architecture.

We can send many different types of events to Amazon EventBridge, which is a serverless event bus service released in July 2019, that allows you to take actions based on the event received. Given its ability to ingest many different types of events, our customers use EventBridge in many ways. Some customers capture events from individual AWS accounts and send those events to an EventBridge event bus in a centralized AWS account for monitoring purposes. In this blog post, I’ll show you how to capture Amazon Elastic Compute Cloud (Amazon EC2) launch failure events and build a centralized event-driven monitoring solution using EventBridge and Amazon CloudWatch.

When you send events to an EventBridge event bus in an AWS account, an EventBridge rule is triggered when an event pattern is matched. When the rule is triggered, you can configure EventBridge to invoke up to five targets that include AWS services or an event bus in another account. When you send an event to an event bus in another AWS account, you can configure rules on the receiver account to do something with the event (for example, to send it to a CloudWatch log group).

Figure 1 shows a centralized EventBridge architecture, where you have a view of events that are happening across your AWS accounts or organizations in AWS Organizations. By sending specific events from an AWS account to an EventBridge event bus in another AWS account, you can quickly determine the impact by seeing how many accounts and which services are impacted by an incident. 

The cross-account EventBridge architecture uses two accounts: account A and a centralized account. In account A, a health event points to a default event bus. A rule in account A sends the health event to an event rule target. The target is a custom event bus that resides in the centralized account. The event bus in the centralized account sends the event details through an EventBridge rule to a CloudWatch log group in the centralized account.

Figure 1: Sender and receiver topology

You can have more than one event bus in the centralized EventBridge account and each event bus can be used for a different purpose. For example, you can send events from your AWS dev and test accounts to a different event bus in the centralized account. You can configure different targets on each event bus based on escalation paths or SLAs for your dev and test environments. This approach is helpful for reporting and for understanding which events are impacting your production services versus dev and test services.

Because you are only sending events that you’re interested in, you keep the number of events and data sent to the centralized event bus to a minimum. This helps you reduce costs and the number of log streams in the CloudWatch log groups.

Use case

As a member of a central Site Reliability Engineering (SRE) team, I want to centralize EC2 health events for my dev, test, and production AWS accounts, so I can be alerted when there is a spike in EC2 launch failures across my organization created in AWS Organizations.

Following the architecture in Figure 1, we can capture an EC2 launch failure event on an account, which we’ll call account A, and send this event to an event bus in another AWS account, which we’ll call the centralized account. This centralized AWS account can be part of your organization and owned by your SRE team. Have a read of the Best Practices for Organizational Units with AWS Organizations blog that describes how to set up centralized security accounts in you AWS Organization. In the centralized account, you can configure a rule on your custom event bus and add an account using a target, like sending the event to a CloudWatch log group.

Now, I’ll show you how to set up an event bus with a rule in the centralized account and account A.

Here are the three steps we will follow:

• Create an EventBridge event bus in the centralized account.
• Create an EventBridge rule in account A to send events to the EventBridge event bus in the centralized account.
• Create an EventBridge rule in the centralized account to send events to a CloudWatch log group.

Step 1 – Create an EventBridge event bus in the centralized account

Using the centralized account, open the Amazon EventBridge console. In the left navigation pane, choose Events, and then choose Event Bus. Choose Create event bus. In Event bus detail, for Name, enter production.

Figure 2: Event Bus name

When you create an event bus in your centralized account, you are asked to provide a policy that allows other accounts to put events onto the event bus. Because my accounts are part of an organization in AWS Organizations, I am going to use the default template policy that allows all accounts to connect to my production event bus. Here is an example policy similar to mine:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "allow_all_accounts_from_organization_to_put_events",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "events:PutEvents",
      "Resource": "arn:aws:events:us-east-1:111122223333:event-bus/production",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-yyyyyyyyyy"
        }
      }
    }
  ]
}

Replace the AWS Region and account ID in the resource ARN and the org ID in the condition key with values for the centralized account. After you have created your event, from the Actions menu, choose Start discovery, as shown in Figure 3.

Figure 3: Custom event bus

After you start discovery, the Schema Discovery column will display a status of Started. Make a note of the event bus ARN. You will need it later when you configure a target event bus in account A.

Step 2 – Create an EventBridge rule in account A to send events to the EventBridge event bus in the centralized account

Now that you have set up an event bus in the centralized account, using account A, go to the Amazon EventBridge console. Choose Events, choose Rules, and then choose Create rule. Enter a name and description for the rule, as shown in Figure 4.

Figure 4: Rule name and description

In Event matching pattern, choose Pre-defined pattern by service. For Service provider, choose AWS. For Service name, choose Health. For Event type, choose Specific Health events. Choose Specific service(s) and then choose EC2. Choose Specific event type category(s) and then choose issue. Choose Specific event type code(s) and then choose AWS_EC2_INCREASED_LAUNCH_FAILURES. Leave Any resource selected.

Figure 5 shows the completed selections:

Figure 5: EventBridge event pattern

While you’re configuring this event, spend some time looking at the other available service names and event types. You can also configure EventBridge rules on these events following the same approach.

As you build the event pattern in the console, you are also presented with the pattern that EventBridge will use to match. Here is the event pattern for the example in Figure 5:

{
    "source": ["aws.health"],
    "detail-type": ["AWS Health Event"],
    "detail": {
        "service": ["EC2"],
        "eventTypeCategory": ["issue"],
        "eventTypeCode": ["AWS_EC2_INCREASED_LAUNCH_FAILURES"]
    }
}

To complete the configuration, in Select targets, for Target, choose Event bus in another AWS account. In Event Bus, enter the ARN of the centralized account. In Figure 6, the ARN for the event bus in the centralized account is highlighted in red. Replace the <region> and <account-id> placeholders with your values. Include the name of the event bus you created (production) at the end of the ARN. Choose Create a new role for this specific resource.

Figure 6: EventBridge target configuration

When you choose Create a new role for this specific resource, the following IAM policy is created and attached to the Amazon_EventBridge_Invoke_Event_Bus IAM role. The AWS Region (us-east-1) and account ID (111122223333) will be replaced with the information you provided when you created the target.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "events:PutEvents"
            ],
            "Resource": [
                "arn:aws:events:us-east-1:111122223333:event-bus/production"
            ]
        }
    ]
}

The IAM role is configured with a trust relationship using events.amazonaws.com as a trusted entity. This allows the EventBridge API service to send events to the centralized account.

Step 3 – Create an EventBridge rule in the centralized account to send events to a CloudWatch log group.

Now that the configuration for account A is complete, set up a rule that matches events that come from account A in the centralized account, and then forward these events to a CloudWatch log group.

Using the centralized account, go to the EventBridge console. In the left navigation pane, choose Events and then choose Rules. In Select event bus, choose the production event bus you created earlier.

Figure 7: Production event bus

Choose Create rule and then enter a name and description for the rule, as shown in Figure 8:

Figure 8: Name and description for the rule

In Define pattern, choose Event pattern. Under Event matching pattern, choose Custom pattern and then paste the following into the Event pattern field and choose Save.

{
    "source": ["aws.health"],
    "detail-type": ["AWS Health Event"],
    "detail": {
        "service": ["EC2"],
        "eventTypeCategory": ["issue"],
        "eventTypeCode": ["AWS_EC2_INCREASED_LAUNCH_FAILURES"]
    }
}

The configuration in the console will look like this:

Figure 9: EventBridge event pattern

This gives us the same event pattern configuration we set up in account A, without having to select all the options. You can use the custom pattern to create your own event patterns to match on.

In Select event bus, choose the production event bus you configured earlier.

Figure 10: Select event bus

In Select targets, under Target, choose CloudWatch log group. In Log Group, specify a new log group, as shown in Figure 11:

Figure 11: CloudWatch log group target configuration

Choose Create and that’s it!

When you have an event pattern match, you will see a log entry in your centralized CloudWatch log group, like so:

{
  "version": "0",
  "id": "7bf73129-1428-4cd3-a780-95db273d1602",
  "detail-type": "AWS Health Event",
  "source": "aws.health",
  "account": "123456789012",
  "time": "2020-01-01T00:00:00Z",
  "region": "us-west-2",
  "resources": ["i-abcd1111"],
  "detail": {
    "eventArn": "arn:aws:health:us-east::event/
            AWS_EC2_INSTANCE_STORE_DRIVE_PERFORMANCE_DEGRADED_90353408594353980",
    "service": "EC2",
    "eventTypeCode": "AWS_EC2_INCREASED_LAUNCH_FAILURES",
    "eventTypeCategory": "issue",
    "startTime": "Fri, 01 Jan 2021 00:00:00 GMT",
    "eventDescription": [{
      "language": "en_US",
      "latestDescription": "A description of the event will be provided here"
    }],
    "affectedEntities": [{
      "entityValue": "i-abcd1111",
      "tags": {
        "stage": "prod",
        "app": "my-app"
      }
    }]
  }
}

Now you can run CloudWatch Logs Insights queries and create dashboards based on the result set.

Here is a CloudWatch Logs Insights query to count the number of distinct accounts with the detail.eventTypeCode equal to AWS_EC2_INCREASED_LAUNCH_FAILURES:

fields account
| filter detail.eventTypeCode = "AWS_EC2_INCREASED_LAUNCH_FAILURES"
| stats count_distinct (account)

And here is example output that you can add to the SRE monitoring dashboard in your centralized account:

Figure 12: CloudWatch Logs Insights query result

Your SRE teams can now see any issues at a glance across your organization when accounts are reporting events such as AWS_EC2_INCREASED_LAUNCH_FAILURES. As I mentioned at the start of this post, you can also have a dev or test event bus and send events from your respective accounts to these event buses in your centralized account. For example, you might want to send Amazon Simple Notification Service (Amazon SNS) notifications for production accounts to an SRE team and send the events to a CloudWatch log group. Or you might want to use a custom API to log a service ticket. The choice is up to you, based on the SLAs for your AWS accounts.

Conclusion

In this blog post, I showed you how to configure a cross-account EventBridge event-monitoring solution to capture events in account A and send them to a centralized EventBridge account. This blog post will help you architect a solution so you have visibility to events happening across your AWS accounts. I showed you how to configure a production event bus and sent events captured from account A to this production event bus in a centralized EventBridge account. Lastly, I showed you how to use a target to forward the event to CloudWatch so you could use CloudWatch Logs Insights to query the number of accounts that reported receiving the event.

During the configuration of the rules, you might have noticed that there are many other event rules you can configure for the different types of services offered by AWS. I hope that sparked your imagination to collect other events.

About the author