Update your Amazon CloudWatch dashboards automatically using Amazon EventBridge and AWS Lambda

Amazon CloudWatch lets customers collect monitoring and operational data in the form of logs, metrics, and alarms. This allows for easy visualization and notifications regarding their workload health.

Amazon CloudWatch dashboards are customizable home pages in the CloudWatch console that you can use to monitor your resources in a single view, even those resources that are spread across different AWS regions or on-premises. You can use CloudWatch dashboards to create customized views of the metrics and alarms for your resources. Users can add various widgets, such as line, stacked area, number, bar, pie, or text to their dashboard to fit their visualization needs.

A widget can contain information from multiple sources and metrics, such as CPU utilization from multiple Amazon Elastic Compute Cloud (EC2) instances or network traffic statistics from the same EC2 instances. Although creating dashboards and widgets is straight forward, it can require a maintenance effort to update the resource sources if the content is based on the dynamic information, such as EC2 instances that are created or removed during scale-out and scale-in events in the Auto Scaling group. This post will show how to create and update an event-based, automatically-updated CloudWatch dashboard.

For this example, we will provide an overview of how to create and update a CloudWatch dashboard based on the scale-out and scale-in events of an Auto Scaling group. However, this example only scratches the surface of what is possible with the CloudWatch.

Solutions Overview

This solution will use Amazon EC2 Auto Scaling instance lifecycle events and Amazon EventBridge to trigger an AWS Lambda function that will automatically create or update a CloudWatch dashboard, as seen in figure one.

Figure 1: Solution overview, showing the event flow between AutoScaling Group, Amazon EventBridge, AWS Lambda and Amazon CloudWatch

Services used in this solution

Amazon CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, and then visualizes it using automated dashboards so that you can get a unified view of your AWS resources, applications, and services that run in AWS and on-premises.

Amazon EventBridge is a serverless event bus that makes it easier to build event-driven applications at scale using events generated from your applications, integrated Software-as-a-Service (SaaS) applications, and AWS services.

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and SaaS applications, and only pay for what you use.

Getting started

In this post, we will first prepare an environment for our sample, which includes:

• One VPC
• Two subnets
• An Auto Scaling group
• An empty Lambda function (which we will manually edit later)
• An AWS Identity and Access Management role

Then, we will create an EventBridge rule, modify the Lambda function and roles for our sample, and, once the components are in place, we will scale our Auto Scaling group to trigger the dashboard creation and update.

We must highlight that this VPC and other resources are not intended for production use, but merely as an example that you can optionally deploy if you wish to recreate this approach for demonstration purposes. We have a curated set of best practices available in the AWS Well-Architected Framework, and we encourage our customers to incorporate these into their release lifecycle. However, to expedite things, we are providing an AWS CloudFormation template that creates this infrastructure for us.

Cost

If operated for an entire month, this sample CloudFormation stack would cost approximately $0.50 per day, though this may vary based on slight differences in regional prices. Without the CloudFormation components, operating only the automatic dashboard updating resources (including the Lambda function, EventBridge rule, and dashboard) will typically cost between a few cents if the free tier is leveraged, or up to $3 per month if more than three dashboards are created per account.

Execute CloudFormation

The following is the CloudFormation stack that we will be using in this example. Note that this sample is only meant to demonstrate using an Auto Scaling group to trigger dynamic dashboard updates, and it is not intended for production use. We use this as a way of rapidly creating an environment that demonstrates the solution. If you are unfamiliar with AWS CloudFormation and stacks, then please see our documentation here, or this video introduction.

AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation Template that creates VPC, ASG, IAM roles for Sample environment
Parameters:
  LambdaName:
    Type: String
    Default: WorkshopLambdaFunction
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
Resources:
  WorkshopVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 10.100.0.0/16
      Tags:
        - Key: Application
          Value: !Ref 'AWS::StackId'
        - Key: Name
          Value: Workshop_VPC
  WorkshopSubnet1:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref WorkshopVPC
      CidrBlock: 10.100.0.0/24
      Tags:
        - Key: Application
          Value: !Ref 'AWS::StackId'
        - Key: Name
          Value: WorkshopSubnet1
  WorkshopSubnet2:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref WorkshopVPC
      CidrBlock: 10.100.1.0/24
      Tags:
        - Key: Application
          Value: !Ref 'AWS::StackId'
        - Key: Name
          Value: WorkshopSubnet2
  WorkshopLaunchTemplate:
    Type: 'AWS::EC2::LaunchTemplate'
    Properties:
      LaunchTemplateName: WorkshopLaunchTemplate
      LaunchTemplateData:
        ImageId: !Ref LatestAmiId
        InstanceType: t3.micro
  WorkshopASG:
    Type: 'AWS::AutoScaling::AutoScalingGroup'
    Properties:
      MinSize: 1
      MaxSize: 3
      DesiredCapacity: 1
      LaunchTemplate:
        LaunchTemplateId: !Ref WorkshopLaunchTemplate
        Version: 1
      VPCZoneIdentifier:
        - !Ref WorkshopSubnet1
        - !Ref WorkshopSubnet2
  WorkshopLambdaRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: WorkshopLambdaRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: '/'
  WorkshopLambdaRolePolicy:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: WorkshopLambdaRolePolicy
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 'ec2:DescribeInstances'
            Resource: '*'
          - Effect: Allow
            Action:
              - 'cloudwatch:PutDashboard'
              - 'cloudwatch:DeleteDashboards'
            Resource: !Sub 'arn:aws:cloudwatch::${AWS::AccountId}:dashboard/*'
          - Effect: Allow
            Action: 'logs:CreateLogGroup'
            Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:*'
          - Effect: Allow
            Action:
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
            Resource:
              Fn::Join:
                - ''
                - - 'arn:aws:logs:'
                  - !Ref 'AWS::Region'
                  - ':'
                  - !Ref 'AWS::AccountId'
                  - ':log-group:/aws/lambda/'
                  - !Ref 'LambdaName'
                  - ':*'
      Roles:
        - Ref: WorkshopLambdaRole

  WorkshopLambdaFuction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      FunctionName: !Ref LambdaName
      Role: !GetAtt WorkshopLambdaRole.Arn
      Handler: index.lambda_handler
      Code:
        ZipFile: |
          import json
          def lambda_handler(event, context):
            return {
              'statusCode' : 200,
              'body': json.dumps("completed")
            }
      Description: Invoke a function during stack creation.

With this deployed, your environment will now have the following components in it:

Figure 2: Architecture diagram of resources deployed by the CloudFormation stack

Create EventBridge Rule

With the working environment created, we will now walk you through the process of automating CloudWatch dashboard updates.

First, select here to launch the Amazon EventBridge console. And then, select Create rule and provide a name for the rule.

Now, select the following fields, in this order:

1. Event pattern
2. Pre-defined pattern by service
3. Service provider: AWS
4. Service Name: Auto Scaling
5. Event Type: Instance Launch and Terminate
6. Any instance event
7. Specific group name(s)

In the drop-down for Specific group names, select the Auto Scaling group created by CloudFormation. Your page should now look similar to the following:

Figure 3: Amazon EventBridge configuration

Leave the Select event bus section untouched, and move on to the Select targets section. Now, select Lambda function, and then WorkshopLambdaFunction.

Figure 4: EventBridge Configuration – select target

Finally, select Create.

Modify Lambda function

Now that we have a Lambda function that will execute every time our Auto Scaling group changes its size, we can add the code to it that actually creates (or recreates) a dashboard. This will be a simple example that creates a single widget with CPU metrics from all instances in our sample Auto Scaling group.

First, go to the Lambda console and select the region where you launched the CloudFormation template. Go into the function created by CloudFormation and select the Configuration tab, where you should see an EventBridge under the Triggers, as seen in the following:

Figure 5: AWS Lambda configuration

Copy and paste the following code into to the editor under the Code tab, in the index.py file.

import copy
import datetime
import json
import os
import boto3


WIDGET_TEMPLATE = {
    'type': 'metric',
    'x': 0,
    'y': 0,
    'width': 15,
    'height': 6,
    'properties': {
        'view': 'timeSeries',
        'stacked': False,
        'metrics': [],
        'region': os.getenv('AWS_DEFAULT_REGION', 'us-east-1'),
        'annotations': {}
    }
}


class CloudWatch:
    """Base class for updated dashboard widgets"""
    def __init__(self, dashboard_name):
        """Name of dashboard to (re)create, list of account numbers to interrogate"""
        self.cw_client = boto3.client('cloudwatch')
        self.ec2_client = boto3.client('ec2')
        self.dashboard_name = dashboard_name
        self.dashboard = None

    def get_ec2_cpu_metrics(self):
        """Returns the EC2 CPU metrics for a single account and region"""
        instances = []
        # CloudWatch metrics can only display max 500 metrics per widget
        results = self.ec2_client.describe_instances(MaxResults=500)
        for result in results['Reservations']:
            for instance in result['Instances']:
                if instance['State']['Code'] != 48:
                    print('adding instance ID: {}'.format(instance['InstanceId']))
                    instances.append(instance['InstanceId'])
                else:
                    print('skipping terminated instance ID: {}'.format(instance['InstanceId']))
        return instances

    def put_dashboard(self, dashboard_body):
        """Puts the updated dashboard into CloudWatch"""
        results = self.cw_client.put_dashboard(
            DashboardName=self.dashboard_name,
            DashboardBody=json.dumps(dashboard_body))
        print(results)


def format_widget(list_of_instance_ids):
    """
    Returns a JSON object with the widget definition for a single account and
    its EC2 instances
    """
    results = []
    for instance in list_of_instance_ids:
        results.append(['AWS/EC2', 'CPUUtilization', 'InstanceId', instance])
    return results


def lambda_handler(event, context):
    """Entrypoint for the Lambda function"""
    cw = CloudWatch(os.getenv('DASHBOARD_NAME'))
    final_dashboard_body = {'widgets': []}
    all_instances = cw.get_ec2_cpu_metrics()
    metrics_list = format_widget(all_instances)
    widget = copy.deepcopy(WIDGET_TEMPLATE)
    widget['properties']['metrics'] = metrics_list
    widget['properties']['title'] = 'CPU Utilization: {} instances'.format(str(len(all_instances)))
    widget['properties']['annotations']['vertical'] = [
        {'label': 'Last updated', 'value': datetime.datetime.utcnow().isoformat() + 'Z'}]
    final_dashboard_body['widgets'].append(widget)
    cw.put_dashboard(final_dashboard_body)

Then, select Deploy to push your changes to become live.

One final change to our Lambda function is required. You must enter an environment variable that will determine the name of the new dashboard.

Note that this script will overwrite any existing dashboard of the same name. Make sure that you don’t use the name of an existing dashboard, as you will then lose it inadvertently.

Select Configuration, and then Environment variables. Now, select Edit to create a new variable. The name must be DASHBOARD_NAME, and the value is the name that you choose.

Figure 6: AWS Lambda function environment variables

A note about proper IAM permissions

When we created our Lambda function with our CloudFormation template, we also created an execution role that gives the Lambda function the basic permission to write logs to CloudWatch Logs and associated it with the function. In order for our code to work properly, it requires the following permissions:

1. Describe the Auto Scaling group and retrieve the instance details that will be added to the dashboard.
2. Execute the CloudWatch PutDashboardAPI endpoint that creates the dashboard (for demo purpose, this IAM policy was created with the CloudFormation template).

We have added these permissions for you already, but if you are not familiar with the process, then you can view these privileges from the Lambda function’s Permissions view under the Configuration tab. Your view will appear similar to the following:

Figure 7: Execution role for our Lambda function

Verification

Now you have all of the pieces needed to create the dashboard based on the events from the Auto Scaling group. Let’s go ahead and trigger it by adjusting our deployment.

The Auto Scaling group created by the CloudFormation template has a desired capacity of one, so we should see exactly one instance has been created under the Auto Scaling group. Open your Auto Scaling group console here, and go to the work shop Auto Scaling Group. Select Edit in the Group details section, and change the desired capacity to two.

Figure 8: Auto Scaling Group size

You can monitor the Lambda Execution through the Monitor tab in the Lambda function, or you can check the logs from the CloudWatch logs.

Check your dashboard

Now, go to the CloudWatch Dashboards (or reload your CloudWatch dashboard page). Your newly updated dashboard will be ready for you!

Figure 9: Auto Scaling Group dashboard sample

Summary

The example in this post shows how you can automatically create or modify the Cloud Watch Dashboard based on the lifecycle events from EventBridge. This solution will be handy when you must manage much more complicated situations with multiple Auto Scaling groups and many instances with the least effort.

Cleanup

To cleanup resources used in this post, follow these steps:

1. Delete the CloudFormation stack created earlier
2. Delete the CloudWatch log group created by your CloudFormation stack’s Lambda function (it will have a name that includes the stack’s name)
3. Delete the EventBridge rule that triggers your updates
4. Delete the generated dashboard from CloudWatch

Next steps

This post only scratches the surface of what is possible with CloudWatch dashboards and you can extend this solution to include more EC2 metrics, data from Amazon Elastic Block Store, Amazon Kinesis, Amazon API Gateway, or any of our services that emit metric data into CloudWatch. You can even perform more advanced automation including custom widgets, CloudWatch logs, and more.

About the authors