AWS Cloud Operations Blog

Manage Amazon CloudWatch agent deployment at scale using the AWS Cloud Development Kit to optimize AWS usage

In this blog post, we will show you how you can programmatically deploy the Amazon CloudWatch agent using the AWS Cloud Development Kit (AWS CDK) as you create your Amazon Elastic Compute Cloud (Amazon EC2) instances. You can use the command line, AWS Systems Manager, and AWS CloudFormation to install the CloudWatch agent on your EC2 instances. We also recently announced that the CloudWatch agent is open source and included with Amazon Linux 2, where it can easily be installed through the yum package manager.

Rather than use these methods, many customers prefer to work with familiar programming languages to handle their Infrastructure as Code (IaC) tasks, including the deployment of the CloudWatch agent as their EC2 instances are created. In this post, we show how to deploy the CloudWatch agent on Amazon Linux 2 instances and Red Hat Enterprise Linux 8.2.

Amazon CloudWatch Agent

Many customers rely on Amazon CloudWatch for the data they need to monitor their AWS services, optimize their compute costs, and operate efficiently as they scale. The CloudWatch agent helps by providing metrics that are not collected by default (for example, memory, swap, disk usage, disk I/O, process details, and more). Custom application metrics are also available through integrations such as CollectD and StatsD.

These metrics can be used for traditional monitoring, alerting, and troubleshooting tasks and by cost-optimization tools such as the AWS Compute Optimizer, which provides money-saving recommendations based on CloudWatch compute and memory metrics.

AWS Cloud Development Kit

Provisioning cloud applications at scale can be a challenging process that requires you to perform manual actions, write custom scripts, maintain templates, or learn domain-specific languages. To simplify this, AWS CDK uses the familiarity and expressive power of programming languages for modeling your applications. It provides you with high-level components called constructs that preconfigure cloud resources with proven defaults, so you can build cloud applications without needing to be an expert. AWS CDK provisions your resources in a safe, repeatable manner through AWS CloudFormation. It also enables you to compose and share your own custom constructs that incorporate your organization’s requirements, helping you start new projects faster. For more information, see Getting started with the AWS CDK in the AWS CDK developer guide.

AWS Compute Optimizer

By using machine learning to analyze historical utilization metrics, AWS Compute Optimizer recommends optimal AWS resources for your workloads to reduce costs and improve performance. Overprovisioning resources can lead to unnecessary infrastructure costs, while under provisioning resources can lead to poor application performance. Compute Optimizer helps you choose the optimal EC2 instance types, including those that are part of an Amazon EC2 Auto Scaling group, and optimal Amazon EBS volume configurations, based on your utilization data.

Add the CloudWatch Agent to your IaC solution with AWS CDK on Amazon Linux 2

Note: If you want an RHEL8 instance and not an Amazon Linux 2 instance, skip this section and go to “Add the CloudWatch Agent to your IaC solution with AWS CDK on Red Hat Enterprise Linux 8.”

Prerequisites

First, set up your development environment. AWS Cloud9 lets you write, run, and debug your code with a browser. It includes all the prerequisites for using the CDK. Follow the steps to create an EC2 environment. On the Configure settings page, for Environment type, choose Create a new EC2 instance for environment (direct access).  For Platform, choose Amazon Linux 2. Choose Open IDE to connect to your environment and you are ready to begin!

Note: If you want to use your own development environment, follow these instructions to install the prerequisites. First, make sure that your development workstation has the AWS Command Line Interface (AWS CLI) installed, that the CLI has been configured correctly, and that your workstation meets the prerequisites for the CDK Toolkit. Our example uses Python, so make sure you have downloaded and installed version 3.6 or later (including pip and virtualenv). Also confirm that the operating system of the EC2 instance on which you intend to install the CloudWatch agent is supported. In our example, we use Amazon Linux 2.

Step 1: Create the application

The following example commands are in bash.

$ mkdir cdkproject

$ cd cdkproject

$ cdk init sample-app --language python

The commands create an empty directory named cdkproject on your workstation and use the CLI to initialize the sample app in Python.

In addition to the code that this command unpacks, it also creates a virtual environment in your directory. This gives you a self-contained, isolated environment where you can run Python and install test packages without polluting your system Python.

To activate the virtual environment on a Linux or macOS platform and install the required Python modules, enter the following:

$ source .venv/bin/activate

(.env) $ pip install -r requirements.txt

The first time you deploy an AWS CDK app into an environment (the combination of your AWS account and AWS Region), you must install a bootstrap stack. This stack includes resources that are required for the operation of the toolkit. For example, the stack includes an Amazon Simple Storage Service (Amazon S3) bucket that is used to store templates and assets during the deployment process.

To install the bootstrap stack, issue the following command:

$ cdk bootstrap aws://<accountnumber>/<region>
Example:
$ cdk bootstrap aws://123456789012/us-east-1

These steps are standard for any Python CDK project. Now you’re ready to put together the components required to deploy two instances with different operating systems with the CloudWatch agent installed on each.

Step 2: Create CloudWatch Agent installer for Amazon Linux 2

In the cdkproject directory, create a file named configure-al2.sh and copy and paste the following script into it. This is the user data script that runs at the EC2 instance creation time for the Amazon Linux 2 instance.

#!/bin/sh
# Use this to install software packages
echo "Userdata script did run" >> /tmp/script_confirmation.txt
yum install -y amazon-cloudwatch-agent
amazon-cloudwatch-agent-ctl -a start

The AWS Cloud9 IDE displays the AWS CDK directory structure. The configure-al2.sh file is displayed in the left pane

Figure 1: Creating the configure-al2.sh file in the AWS CDK directory structure

Step 3: Use the CDK to deploy Amazon Linux 2 with the CloudWatch Agent installed

The cdkproject directory contains a file named app.py. Using the AWS Cloud9 text editor (or any editor of your choice), replace the contents of this file with the following script. Check the comments in the surrounding code for the items that must be replaced with specifics from your AWS account.

Step 3.1: Import the required CDK libraries

#!/usr/bin/env python3

import os.path

from aws_cdk.aws_s3_assets import Asset
from aws_cdk.core import App, Stack, Environment
from aws_cdk import (
    aws_ec2 as ec2,
    aws_iam as iam,
    core
 )

Step 3.2: Assign variables

Replace all instances of the word VALUE to match your environment. Check descriptions and examples in the comments.

# Variable defintions

# Your AWS Region
# ex: my_region = “us-east-1”
my_region = “VALUE”

# Your AWS account number
# ex: my_account_num = “123456789012”
my_account_num = "VALUE"

# The ID of the Virtual Private Cloud the new instance should use
# ex: my_vpc_id = “vpc-abcdef123456”
my_vpc_id = "VALUE"

# The ID of the Security Group the new instance should use
# ex: my_sg_id = “"sg-abcdef123456"
my_sg_id = "VALUE"

# The instance type for your new instance.  
# t2.micro is good for this demo, but if you want to upgrade you can do so.
my_instance_type = "t2.micro"

# The name of the security key pair that should be authorized to SSH into the new instance.
# These can be found or created in the console at EC2 → key pairs.
# ex: my_key_name = “devtestkey”
my_key_name = "VALUE"

Step 3.3: Define the CDK class to create EC2 instance stack

dirname = os.path.dirname(__file__)

class EC2InstanceStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

Step 3.4:  Assign the VPC for the new EC2 instance

This is assigned to parameter vpc.

vpc = ec2.Vpc.from_lookup(self, "VPC",
    vpc_id = my_vpc_id
)

Step 3.5: Assign the security group for the new EC2 instance

This is assigned to parameter sg.

sg = ec2.SecurityGroup.from_security_group_id(self, "SG", my_sg_id, mutable=False)

Step 3.6: Set IAM requirements (instance role and System Manager managed policy)

This is assigned to parameter role.

role = iam.Role(self, "InstanceSSM", assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))
role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AmazonEC2RoleforSSM"))

Step 3.7: Set up the AL2 instance details

The CDK can reference the latest AL2 AMI as latest_amazon_linux. This is assigned to parameter amzn_linux.

amzn_linux = ec2.MachineImage.latest_amazon_linux(
# These settings and more can be configured for the new AL2 instance
       generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
       edition=ec2.AmazonLinuxEdition.STANDARD,
       virtualization=ec2.AmazonLinuxVirt.HVM,
       storage=ec2.AmazonLinuxStorage.GENERAL_PURPOSE
 )

Combine the preceding parameters into an instance parameter and name it al2_instance.

al2_instance = ec2.Instance(self, "AL2-Instance",
        instance_type=ec2.InstanceType("t2.micro"),
        machine_image=amzn_linux,
        key_name = my_key_name,
        vpc = vpc,
        role = role,
        security_group = sg
  )

Add the configure-al2.sh script file to the S3 bucket set up by CDK. Type asset tells the CDK to upload this file to S3. It is then retrieved and run as part of user_data.

asset_al2 = Asset(self, "userdata-al2", path=os.path.join(dirname, "configure-al2.sh"))
local_path_al2 = al2_instance.user_data.add_s3_download_command(
        bucket=asset_al2.bucket,
        bucket_key=asset_al2.s3_object_key
)

Add the previously uploaded asset (configure-al2.sh) to user_data. This causes the script to run on instance creation.

al2_instance.user_data.add_execute_file_command(
        file_path=local_path_al2
)
asset_al2.grant_read(al2_instance.role)

Step 3.8: Create the EC2 instance

app = core.App()
EC2InstanceStack(app, "ec2-cdk-instances", env=core.Environment(account=my_account_num, region=my_region))
app.synth()

Step 4: Deploy the instance

Use the following command:

$ cdk deploy ec2-cdk-instances

If you are prompted to deploy the changes or to change IAM policies for Systems Manager (SSM), select the option y.

Add the CloudWatch Agent to your IaC solution with AWS CDK on Red Hat Enterprise Linux 8

Note: If you want an Amazon Linux 2 instance and not an RHEL8 instance, you can skip this section and go to “Confirm the CloudWatch Agent was installed successfully on the new instance.”  If you want to set up both the AL2 and RHEL8 instance types and you have already set up AL2, go to Step 3.2, add just the last variable, then skip to Step 3.7, and then skip to Step 4.

Prerequisites

First, set up your development environment. AWS Cloud9 lets you write, run, and debug your code with a browser. It includes all the prerequisites for using the CDK. Follow the steps to create an EC2 environment. On the Configure settings page, for Environment type, choose Create a new EC2 instance for environment (direct access).  For Platform, choose Amazon Linux 2. Choose Open IDE to connect to your environment and you are ready to begin!

Note: If you want to use your own development environment, follow these instructions to install the prerequisites. First, make sure that your development workstation has the AWS CLI installed, that the CLI has been configured correctly, and that your workstation meets the prerequisites for the CDK Toolkit. Our example uses Python, so make sure you have downloaded and installed version 3.6 or later (including pip and virtualenv). Also confirm that the operating system of the EC2 instance on which you intend to install the CloudWatch agent is supported. In our example, we use Red Hat Enterprise Linux 8.2.

Step 1: Create the application

The following example commands are in bash.

$ mkdir cdkproject

$ cd cdkproject

$ cdk init sample-app --language python

The commands create an empty directory named cdkproject on your workstation and use the CLI to initialize the sample app in Python.

In addition to the code that this command unpacks, it also creates a virtual environment in your directory. This gives you a self-contained, isolated environment where you can run Python and install test packages without polluting your system Python.

To activate the virtual environment on a Linux or macOS platform and install the required Python modules, enter the following:

$ source .venv/bin/activate

(.env) $ pip install -r requirements.txt

The first time you deploy an AWS CDK app into an environment (the combination of your AWS account and chosen AWS Region), you must install a bootstrap stack. This stack includes resources that are required for the toolkit’s operation. For example, the stack includes an S3 bucket that is used to store templates and assets during the deployment process.

To install the bootstrap pack, issue the command:

$ cdk bootstrap aws://<accountnumber>/<region>
Example:
$ cdk bootstrap aws://123456789012/us-east-1

The preceding steps are standard for any Python CDK project. Now you’re ready to put together the components required to deploy two instances with different operating systems with the CloudWatch agent installed on each.

Step 2: Install CloudWatch Agent in Red Hat Enterprise Linux

In the cdkproject directory, create a file named configure-rhel.sh and copy and paste the following script into it. This is the user data script that runs at the EC2 instance creation time for the Red Hat Enterprise Linux instance.

#!/bin/sh
# Use this to install software packages
echo "Userdata script did run" >> /tmp/script_confirmation.txt
sudo dnf install -y https://s3.us-east-1.amazonaws.com/amazon-ssm-us-east-1/latest/linux_amd64/amazon-ssm-agent.rpm
rpm -Uvh https://s3.amazonaws.com/amazoncloudwatch-agent/redhat/amd64/latest/amazon-cloudwatch-agent.rpm
amazon-cloudwatch-agent-ctl -a start
sudo systemctl restart amazon-ssm-agent

Figure 2: Creating the configure-rhel.sh file in the AWS CDK directory structure

Step 3: Use the CDK to deploy RHEL8 with the CloudWatch Agent installed

The cdkproject directory contains a file named app.py. Using the AWS Cloud9 text editor (or any editor of your choice), replace the contents of this file with the following script. See the comments in the surrounding code for the items that must be replaced with specifics from your AWS account.

Step 3.1: Import the required CDK libraries

#!/usr/bin/env python3

import os.path

from aws_cdk.aws_s3_assets import Asset
from aws_cdk.core import App, Stack, Environment
from aws_cdk import (
    aws_ec2 as ec2,
    aws_iam as iam,
    core
 )

Step 3.2: Assign variables

Replace all instances of the word VALUE to match your environment. See descriptions and examples in the comments.

# Variable defintions

# Your AWS Region
# ex: my_region = “us-east-1”
my_region = “VALUE”

# Your AWS account number
# ex: my_account_num = “123456789012”
my_account_num = "VALUE"

# The ID of the Virtual Private Cloud the new instance should use
# ex: my_vpc_id = “vpc-abcdef123456”
my_vpc_id = "VALUE"

# The ID of the Security Group the new instance should use
# ex: my_sg_id = “"sg-abcdef123456"
my_sg_id = "VALUE"

# The instance type for your new instance.  
# t2.micro is good for this demo, but if you want to upgrade you can do so.
my_instance_type = "t2.micro"

# The name of the security key pair that should be authorized to SSH into the new instance.
# These can be found or created in the console at EC2 → key pairs.
# ex: my_key_name = “devtestkey”
my_key_name = "VALUE"

# For RHEL and all other instance types other than Amazon Linux 2, specify the AMI ID.  
# ex: my_ami_id = “ami-096fda3c22c1c990a”
my_ami_id = "VALUE"

Note: The best way to find the AMI ID for your Region/OS combination is to start creating an EC2 instance in the Amazon EC2 console, using your AWS Region.

On the Choose an Amazon Machine Image page, the words “Red Hat” are entered into the search box, which returns Red Hat Enterprise Linux 8 (HVM), SSD Volume Type.

Figure 3: Locating the AMI ID of your AMI

Step 3.3: Define the CDK class to create the EC2 instance stack

dirname = os.path.dirname(__file__)

class EC2InstanceStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

Step 3.4:  Assign the VPC for the new EC2 instance

This is assigned to parameter vpc.

vpc = ec2.Vpc.from_lookup(self, "VPC",
    vpc_id = my_vpc_id
)

Step 3.5: Assign the security group for the new EC2 instance

This is assigned to parameter sg.

sg = ec2.SecurityGroup.from_security_group_id(self, "SG", 
my_sg_id, mutable=False)

Step 3.6: Set IAM requirements (instance role and SSM managed policy)

This is assigned to parameter role.

role = iam.Role(self, "InstanceSSM", assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))
role.add_managed_policy(iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AmazonEC2RoleforSSM"))

Step 3.7: Set up the RHEL8 instance details

For RHEL (or other OS variants), use generic types.

rhel_linux = ec2.MachineImage.generic_linux({
    my_region: my_ami_id
    })

Combine the preceding RHEL parameters into an instance parameter and name it rhel_instance.

rhel_instance = ec2.Instance(self, "RHEL-Instance",
    instance_type=ec2.InstanceType("t2.micro"),
    machine_image=rhel_linux,
    #Set this to the name of a previously stored key pair for ssh
    key_name = "CHANGEME",
    # example: key_name = "my-key-pair"
    vpc = vpc,
    role = role,
    security_group = sg
    )

For instance types other than AL2, you must add the AWS CLI to the instance using the following commands.

    rhel_instance.user_data.add_commands(
        'sudo dnf -y install unzip',
        'curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscliv2.zip',
        'unzip /tmp/awscliv2.zip -d /tmp',
        'sudo /tmp/aws/install'
    )

Note: You can use this technique for adding the CLI instead of the asset/S3 technique by simply adding all the commands from configure-rhel.sh here.

Add the configure-rhel.sh script file to the Amazon S3 bucket set up by the CDK. Type asset tells the CDK to upload this file to Amazon S3. It is then retrieved and run as part of user_data.

asset_rhel = Asset(self, "userdata-rhel", path=os.path.join(dirname, "configure-rhel.sh"))
local_path_rhel = rhel_instance.user_data.add_s3_download_command(
    bucket=asset_rhel.bucket,
    bucket_key=asset_rhel.s3_object_key
)

Add the previously uploaded asset (configure-rhel.sh) to user_data. This causes the script to run on instance creation.

rhel_instance.user_data.add_execute_file_command(
    file_path=local_path_rhel
)
asset_rhel.grant_read(rhel_instance.role)

Step 3.8: Create the EC2 instance

app = core.App()
EC2InstanceStack(app, "ec2-cdk-instances", env=core.Environment(account=my_account_num, region=my_region))
app.synth()

Step 4: Deploy the instance

Use the following command:

$ cdk deploy ec2-cdk-instances

If prompted to deploy the changes or to change IAM policies for System Manager (SSM), select y.

Confirm the CloudWatch Agent was installed successfully on the new instance

To confirm that your new instances are running with the CloudWatch agent installed, connect to each instance and issue the following commands:

$ cat /tmp/script_confirmation.txt

$ amazon-cloudwatch-agent-ctl -a status

The status of the CloudWatch agent should be displayed as running. While you are logged into the instance, you can also configure the CloudWatch agent to your specifications, and you will begin to see CloudWatch metrics from your instance available in the AWS Management Console. If you want to use this configuration for other instances you’re provisioning, you can modify the sample code to roll out your CloudWatch agent configuration file with the CloudWatch agent itself.

Integration with Compute Optimizer

Finally, to get the longer-term benefits of these additional metrics, go to the AWS Compute Optimizer in the AWS Management Console. You can create a dashboard in Compute Optimizer to analyze the specifications and utilization metrics of your AWS resources. If this is the first time use of the Compute Optimizer in the supported AWS Region in which you’re working, you’ll need to opt in to authorize the Compute Optimizer to access AWS resources. To learn more about setting up dashboards, see Getting Started with AWS Compute Optimizer.

The dashboard looks similar to the following:

The dashboard displays a Findings per AWS resource search box. The selected Region is US East (N. Virginia). There are tiles on the page for EC2 instances, EBS volumes, and Auto Scaling groups.

Figure 4: Compute Optimizer dashboard

Note: Compute Optimizer requires 30 consecutive hours of CloudWatch metrics on your resources to generate findings and recommendations. Keep your provisioned instances running for that period of time to get the information to appear on the dashboard.

The dashboard compares current compute usage against recommended options along with savings per instance type.

Under Compare current instance type with recommended options, option 1, a t3.micro, is selected.

Figure 5: Compute Optimizer – instance recommendations

You can generate a detailed utilization report that shows CPU, memory, network in/out, and so on. You can also compare metrics to determine the optimal instance type for your workload.

The metrics displayed on the page are CPU utilization, Memory utilization, Network in, Network out, Amazon EBS read throughput, EBS write throughput, Amazon EBS read bandwidth, and EBS write bandwidth.

Figure 6: Detailed compute utilization metrics in Compute Optimizer

The dashboard also provides recommendations on optimal EBS volume (Current IOPS, Recommended IOPS), including expected savings opportunity and estimating impact of using recommended configurations

Under Current resource utilization, there are graphs for read throughput (operations/second), write throughput (operations/second), read bandwidth (KiB/second), and write bandwidth (KiB/second).

Figure 7: Detailed storage utilization metrics in Compute Optimizer

Cleanup

The preceding code provisions EC2 instances in your account. To avoid ongoing charges, make sure to remove them after running your test. Run the following command to use the CDK to delete provisioned resources:

$ cdk destroy ec2-cdk-instances

Check the AWS CloudFormation console to confirm that the CloudFormation stack was successfully deleted by the CDK. Add hyperlink to delete stack from AWS documentation for better readability.

Delete the Cloud9 environment created above.

Conclusion

In this post, we introduced you to the AWS Cloud Development Kit as a tool for installing the CloudWatch agent on EC2 instances of different operations systems. We showed how you can write CDK scripts using Python to install the CloudWatch agent.

AWS CDK gives you the expressive power of programming languages for defining infrastructure. You can write your runtime code and define your AWS resources with the same programming language. To help get you started, see the following resources:

·      AWS CDK developer guide

·      AWS CDK reference documentation

·      Python CDK code examples for fully functional applications

About the authors

 

Jeff Strickland is a Technical Account Manager with Amazon Web Services (AWS), with 20+ years of expertise in Systems Engineering, Automation, and DevOps Evangelism.

 

 

 

 

Neel Sendas is a Senior Technical Account Manager at Amazon Web Services. Neel works with enterprise customers to design, deploy, and scale cloud applications to achieve their business goals. He is also a machine learning enthusiast. When he is not helping customers, he dabbles in golf and salsa dance.

 

Chris Spruell is a Senior Solutions Architect with Amazon Web Services (AWS). During his 25+ years in technology, he has worked with customers of all sizes to build resilient, secure, and operationally efficient architectures for their enterprise workloads.