Viewing custom metrics from statsd with Amazon Managed Service for Prometheus and Amazon Managed Grafana

sudo apt-get update
curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt-get install -y nodejs
git clone https://github.com/statsd/statsd
cd statsd
npm i

curl -sOL https://github.com/prometheus/statsd_exporter/releases/download/v0.22.4/statsd_exporter-0.22.4.linux-amd64.tar.gz
tar -xvf statsd_exporter-0.22.4.linux-amd64.tar.gz

Configure statsd_exporter

The statsd_exporter can be configured to translate specific dot-separated statsd metrics into labelled Prometheus metrics via a simple mapping language. This mapping is saved in a YAML file and used for launching statsd_exporter.

1. Open a new file named statsd-mapping-config.yaml in the statsd_exporter-0.22.4.linux-amd64 folder.
2. Copy and paste the following into statsd-mapping-config.yaml

mappings:
- match: "api.*.*.*"
  name: "api_metrics"
  labels:
    path: "$1"
    method: "$2"
    status: "$3"

This mapping file will take a metric named, for example, api.hello.GET.200, and transform it into the Prometheus format as api_metrics{path=hello,method=GET,status=200}. The metric api.hello.GET.200 will be generated by the NodeJs application that has a statsd client.

Launch Amazon Managed Service for Prometheus workspace

1. Create Amazon Managed Service for Prometheus workspace as described here.
2. Copy the workspace ID of the created workspace as described here. This workspace ID will be used in the section for configuring the Prometheus server.
3. Create an IAM role with AmazonPrometheusRemoteWriteAccess policy and attach this role to the Amazon EC2 instance.
4. Add the following into the YAML file named statsd-prometheus.yaml, and upload it to the created workspace. This will generate an average value of API invocations over a 5 minute duration based on the path and status code.

groups:
  - name: statsd
    rules:
    - record: metric:api_metrics
      expr: sum(rate(api_metrics[5m])) by (path, status)

Install Prometheus server

1. Open a new terminal window.
2. Connect to your Linux instance using Session Manager.
3. Run the following commands to install Prometheus server.

export PROMETHEUS_VERSION=2.34.0
curl -OL https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}-rc.0/prometheus-${PROMETHEUS_VERSION}-rc.0.linux-amd64.tar.gz
tar xvzf prometheus-${PROMETHEUS_VERSION}-rc.0.linux-amd64.tar.gz
export PATH=$PATH:$HOME/prometheus-${PROMETHEUS_VERSION}-rc.0.linux-amd64

Configure Prometheus server

1. Create a configuration file named prometheus.yaml using the following content. Change regionId as applicable. Change workspaceId with the workspace ID from the section on launching Amazon Managed Service for Prometheus workspace.

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'statsd_exporter'
scrape_configs:
  - job_name: 'statsd_exporter'
    static_configs:
      - targets: ['localhost:9102']
remote_write:
  -
    url: https://aps-workspaces.<regionId>.amazonaws.com/workspaces/<workspace Id>/api/v1/remote_write
    queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500
    sigv4:
        region: <regionId>

Launch Prometheus server

1. Open a new terminal window.
2. Connect to your Linux instance using Session Manager.
3. Launch the Prometheus server with the prometheus.yaml configuration file created previously using the following commands.

export PROMETHEUS_VERSION=2.34.0
export PATH=$PATH:$HOME/prometheus-${PROMETHEUS_VERSION}-rc.0.linux-amd64
prometheus --config.file=prometheus.yaml 

The console should show output similar to this to indicate that Amazon Managed Service for Prometheus was connected to by the Prometheus server successfully:

ts=2022-03-04T01:12:25.394Z caller=dedupe.go:112 component=remote level=info remote_name=4b1ae2 url=https://aps-workspaces.regionId.amazonaws.com/workspaces/workspaceId/api/v1/remote_write msg="Starting WAL watcher" queue=4b1ae2
ts=2022-03-04T01:12:25.394Z caller=dedupe.go:112 component=remote level=info remote_name=4b1ae2 url=https://aps-workspaces.regionId.amazonaws.com/workspaces/workspaceId/api/v1/remote_write msg="Starting scraped metadata watcher"
ts=2022-03-04T01:12:25.394Z caller=dedupe.go:112 component=remote level=info remote_name=4b1ae2 url=https://aps-workspaces.regionId.amazonaws.com/workspaces/workspaceId/api/v1/remote_write msg="Replaying WAL" queue=4b1ae2

Launch a NodeJs application instrumented with statsd client

1. Open a new terminal window
2. Connect to your Linux instance using Session Manager.
3. Run the following commands to create a folder and initialize:

mkdir app
npm init -y
npm i express node-statsd --save

4.  Open a new file named index.js and paste the following:

const express = require('express')
const StatsD = require('node-statsd')
const statsdClient = new StatsD()
const app = express()
const port = 4000
let incrVar
app.get('/hello', (req, res) => {
  incrVar = `${req.path}.${req.method}.200`
  incrVar = `api.${incrVar.substring(1)}`
  statsdClient.increment(incrVar)
  res.status(200).send('Hello World!')
})
app.use('/', (req, res) => {
  incrVar = `${req.path}.${req.method}.404`
  incrVar = `api.${incrVar.substring(1)}`
  statsdClient.increment(incrVar)
  res.status(404).send('Not found.')
})
app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

This NodeJs application exposes an ExpressJs based endpoint /hello on port 4000 for GET verb. This endpoint responds with a simple text “Hello World!” with status code 200. Invoking any other endpoint will result in a response as “Not found.” with a status code as 404.

This application uses the node-statsd package for implementing the statsd client. The metric emitted will have the format name api.<requestPath>.<requestMethod>.<statusCode> where, the requestPath is the end-point invoked, requestMethod is the HTTP method, and the statusCode is set up in the response. The value of this metric is a monotonically increasing counter to track the number of invocations. For example, when /hello is invoked with GET, a metric will be emitted as api.hello.GET.200 with the value incremented by 1.

Launch Grafana workspace

1. Create a workspace as described here.
2. Define user access as described here.

Add Prometheus data source

1. Add the Amazon Managed Service for Prometheus data source as described here.

Create dashboard

1. Create a new dashboard with the Import option as described here.
2. Upload the following JSON document:

{"__inputs":[],"__requires":[{"type":"grafana","id":"grafana","name":"Grafana","version":"8.2.5"},{"type":"panel","id":"timeseries","name":"Time series","version":""}],"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":[],"type":"dashboard"},"type":"dashboard"}]},"editable":true,"fiscalYearStartMonth":0,"gnetId":null,"graphTooltip":0,"id":null,"links":[],"liveNow":true,"panels":[{"datasource":null,"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"smooth","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"auto","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":19,"w":24,"x":0,"y":0},"id":2,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom"},"tooltip":{"mode":"single"}},"targets":[{"exemplar":true,"expr":"metric:api_metrics","interval":"","legendFormat":"{{path}}-{{status}}","refId":"A"}],"title":"API metrics (rate)","type":"timeseries"}],"refresh":"","schemaVersion":32,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-5m","to":"now"},"timepicker":{},"timezone":"browser","title":"statsd","uid":"5fn7dTY7z","version":9}

Instructions for Repeater use case

As discussed earlier, in the Repeater use case,

1. The statsd process is configured with the repeater backend.
2. The application is not changed.

The following instructions will set up the Repeater use case.

Configure statsd process

The statsd process should be reconfigured with a repeater backend with the following commands:

1. Go to the terminal where statsd is launched.
2. Create a file named statsd-config.js, and paste the following.

{
 backends: [ "./backends/console", "./backends/repeater" ],
 repeater: [ { host: "localhost", port: 9125 } ]
}

The ./backends/console is strictly not required. However, using this backend will show the metrics on the console. The repeater points to the host and port of the statsd_exporter running on the same Amazon EC2 instance with 9125 as the default port number.

Launch statsd process

Launch the statsd process with the following command. Make sure that the path to statsd-config.js is correctly provided.

node stats.js statsd-config.js

Launch statsd_exporter

1. Go to the terminal where statsd_exporter was installed.
2. Launch it with the following command. Make sure that the path to statsd-mapping-config.js is correctly provided.

./statsd_exporter --statsd.mapping-config=statsd-mapping-config.yaml

Simulate HTTP requests

Run the following commands in two different terminals to simulate HTTP GET requests at the /hello end-point to generate the metric for 200 status code and /bye end-point to generate the metric for 404 status code. These commands will run for five minutes. The duration can be changed by setting a different value for the -z flag.

1. Get the public IP address of the running Amazon EC2 instance.
2. Enable in-bound TCP traffic from your IP address on port 4000.

hey -z 5m http://ec2-public-ip:4000/hello
hey -z 5m http://ec2-public-ip:4000/bye

View dashboard

1. Launch the dashboard from the Amazon Managed Grafana console as described here.
2. The following graph is seen. Note the legend at the bottom left for curves corresponding to the API path and status.

Instructions for Relay use case

Configure application

1. Go to the terminal where the application is running.
2. Stop the application with Ctrl-C.
3. Edit index.js by changing the declaration of statsdClient as shown in the following:

const statsdClient = new StatsD({
  host: "localhost",
  port: 9125
})

Launch statsd process

1. In general, to use the Relay use case, the statsd process doesn’t require a change. However, in this post, the configuration was changed to add a repeater back-end that was added in an earlier section. It’s recommended that the repeater backend be removed.

Go to the terminal where the statsd process is running, and edit statsd-config.js with the following content:

{
 backends: [ "./backends/console" ]
}

Launch statsd_exporter

1. Go to the terminal where statsd_exporter was installed.
2. Launch it with the following command. Make sure that the path to statsd-mapping-config.js is correctly provided.

./statsd_exporter --statsd.mapping-config=statsd-mapping-config.yaml --statsd.relay.address=localhost:8125

Now the statsd_exporter is configured to relay statistics to the statsd process.

Simulate HTTP requests

Run the following commands in two different terminals to simulate HTTP GET requests at the /hello end-point to generate the metric for 200 status code and /bye end-point to generate the metric for 404 status code. These commands will run for five minutes. The duration can be changed by setting a different value for the -z flag.

1. Get the public IP address of the running Amazon EC2 instance.
2. Enable in-bound TCP traffic from your IP address on port 4000.

hey -z 5m http://ec2-public-ip:4000/hello
hey -z 5m http://ec2-public-ip:4000/bye

View dashboard

1. Launch the dashboard from the Amazon Managed Service for Grafana console  by clicking on the Grafana workspace URL.
2. The following graph is seen. Note the legend at the bottom left for curves corresponding to the API path and status.

Clean up

To avoid incurring future charges, delete the following resources:

1. Delete Grafana workspace as described here.
2. Delete Prometheus workspace as described here.
3. Terminate the Amazon EC2 instance as described here.
4. Remove the IAM policy as described here.

Conclusion

This post describes a means to ingest, query, and visualize custom metrics from applications collected by the statsd process running on an Amazon EC2 Linux instance using Amazon Managed Service for Prometheus and Amazon Managed Grafana. Two scenarios for using stasd_exporter were discussed – repeater and relay – based on the changes to be done either with statsd or the application. Depending on the requirements, different custom metrics can be created, such as HTTP latency, or other values of the request path, query parameters, etc. Furthermore, Amazon Managed Service for Prometheus and Amazon Managed Grafana can be configured to raise alerts as required.

Author: