Collecting CPU and memory metrics for AWS Lambda MicroVMs

Most production services in AWS use at least two key metrics for service health – CPU and memory utilization. The amount of CPU and memory used by the host (in this case, a MicroVM) can indicate scaling signals or inefficiencies in your application. If you’re running a production workload on AWS Lambda MicroVMs, it’s recommended to have observability in these dimensions. And the easiest way to collect these metrics is through the Amazon CloudWatch Agent.

This blog shows you how to collect CPU and memory metrics from within the MicroVM using the CloudWatch Agent.

How to collect CPU and memory metrics in your MicroVM

To observe how a workload uses CPU and memory over time, run the CloudWatch Agent inside the MicroVM. Since a MicroVM image is a full OS snapshot, you can start the agent during image creation, meaning it will already be running the moment a MicroVM launches from that image. This means zero startup latency and one-time configuration: set up the CloudWatch Agent once in the image, and every MicroVM that launches from it already has a running monitoring stack.

To setup CloudWatch Agent, you will modify the ZIP containing your application and Dockerfile, and build a MicroVM image. Once you run a MicroVM from the image, three metrics will be emitted (cpu_usage_active, cpu_usage_idle, mem_used_percent) under an ImageName dimension populated from a Lambda-injected environment variable.

Lambda-injected environment variables

The Lambda MicroVMs runtime automatically exposes these environment variables to your application:

The example below uses AWS_LAMBDA_MICROVM_IMAGE_NAME as a metric dimension so you can monitor metrics per MicroVM image.

Setting up custom metric dimensions from env variables

Amazon CloudWatch Agent uses telegraf to process metrics and opentelemetry-collector (OTel) to export them. Normally, you configure the agent through a cwagent.json file, which the agent’s config-translator converts into a telegraf TOML file and an OTel YAML file for the process to use at startup.

In this post, we skip the JSON configuration and create the telegraf and OTel files directly. This lets us dynamically set a custom metric dimension from an environment variable using OTel’s ${env:VAR} syntax. The telegraf config defines which metrics to collect, while the OTel config resolves the environment variable at process start and appends it as a dimension.

Configuring CloudWatch Agent

In this section, we cover how to configure CloudWatch Agent to report CPU and memory metrics for MicroVMs launched from your MicroVM image.

Step 1: Configure the telegraf plugin to emit CPU and Memory metrics

Create a cwagent.toml file to define the configuration for telegraf to emit CPU and memory metrics every minute:

[agent]
  interval = "60s"
  flush_interval = "60s"

  # Host name is omitted since it doesn't exist in a MicroVM
  omit_hostname = true

[[inputs.cpu]]
  totalcpu = true

  # Disable per-CPU reporting for an aggregate view over all vCPUs in your MicroVM.
  # Set to 'true' to see utilization for each individual vCPU.
  percpu = false
  report_active = true
  fieldpass = ["usage_active", "usage_idle"]

[[inputs.mem]]
  fieldpass = ["used_percent"]

In this configuration, the chosen metric (used_percent) reports memory usage as a percentage of total memory inside the MicroVM. Telegraf derives this from MemAvailable in /proc/meminfo, which reflects memory that is committed and not reclaimable. When your application releases memory back to the OS (e.g. via free()), that memory becomes reclaimable again, and used_percent decreases accordingly.

To monitor additional memory metrics, you can add the following fields to the fieldpass list:

• cached: for page cache bytes
• buffered: for buffered I/O bytes
• total: for total memory available to the MicroVM

Step 2: Configure OTel to process and export the metrics to CloudWatch

Create a cwagent.yaml file to export metrics to CloudWatch under the namespace LambdaMicroVms/Application with dimension ImageName. The dimension value is populated from the environment variable AWS_LAMBDA_MICROVM_IMAGE_NAME.

receivers:
  telegraf_cpu: { collection_interval: 60s }
  telegraf_mem: { collection_interval: 60s }

processors:
  resource:
    attributes:
      - { key: ImageName, value: "${env:AWS_LAMBDA_MICROVM_IMAGE_NAME}", action: insert }
  transform/strip_cpu_dim:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "cpu")

exporters:
  awscloudwatch:
    namespace: LambdaMicroVms/Application
    region: ${env:AWS_REGION}
    resource_to_telemetry_conversion: { enabled: true }

service:
  pipelines:
    metrics:
      receivers:  [telegraf_cpu, telegraf_mem]
      processors: [resource, transform/strip_cpu_dim]
      exporters:  [awscloudwatch]

If you want more dimensions such as image version, add it to attributes.

Note: since only aggregate CPU usage is emitted by telegraf, we don’t need OTel to include a CPU dimension, so delete_key(attributes, "cpu") is used to remove this dimension.

Step 3: Install CloudWatch Agent in your Dockerfile

In your Dockerfile, install the CloudWatch Agent from the Amazon Linux repository. Then copy over the telegraf and OTel files to where the agent expects to retrieve them. Then configure your application’s entrypoint:

FROM public.ecr.aws/lambda/microvms:al2023-minimal

RUN dnf install -y --setopt=install_weak_deps=0 \
        python3 amazon-cloudwatch-agent \
    && dnf clean all

COPY app.py        /app/app.py
COPY cwagent.toml  /etc/cwagent.toml
COPY cwagent.yaml  /etc/cwagent.yaml
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

CMD ["/entrypoint.sh"]

Step 4: Configure your Entrypoint to start CloudWatch Agent

Create a file called entrypoint.sh to start the CloudWatch Agent as a background process while executing your application in the foreground:

#!/usr/bin/env bash
set -euo pipefail

# Telegraf inputs (TOML) + OTel pipeline (YAML).
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent \
    -config     /etc/cwagent.toml \
    -otelconfig /etc/cwagent.yaml &

exec python3 /app/app.py

This is everything you need to get CloudWatch running inside your MicroVMs!

Execution role requirements

To write the metrics to CloudWatch, ensure the MicroVM’s execution role has cloudwatch:PutMetricData permissions.

Verifying it works

To verify the metrics are being emitted, run the following command a few minutes after launching a MicroVM from your image:

aws cloudwatch list-metrics \
    --namespace LambdaMicroVms/Application \
    --dimensions Name=ImageName,Value=mem-python \
    --region us-west-2

You should see exactly three metric series per image: cpu_usage_active, cpu_usage_idle, and mem_used_percent.

Viewing the metrics

To view the metrics in the CloudWatch console, click “All Metrics”, and select the custom namespace LambdaMicroVms/Application (set in cwagent.yaml namespace field).

Here is an example for how it looks inside the console:

In the graph above, the application consumes ~2% memory (left axis) and < 0.1% CPU usage (right axis) when idle. The application then consumes ~9% of memory at the 30 minute mark, holds it for around 5 minutes, then releases it back to the OS. As it releases memory, we see memory utilization decrease. In this example, the MicroVM size is larger than the application needs – less than 10% of memory was used, indicating a smaller MicroVM size may be more economic for this workload.

If your CPU and/or memory utilization is below the baseline size configured (see MicroVM sizing), consider choosing a lower baseline to reduce your compute bill.

Conclusion

This post shows you how to configure and run the CloudWatch Agent inside your MicroVM image so you can collect CPU and memory metrics for MicroVMs launched from the image. This helps you monitor resource usage of your application as it is used, so you can right-size the MicroVM for your workload, debug service health, and check for scaling signals.

To get started, visit the AWS Lambda console, or install the AWS Lambda MicroVMs agent skill.