Microsoft Workloads on AWS

.NET observability with Amazon CloudWatch and AWS X-Ray: Part 1 — Metrics

Building a well-architected .NET application goes beyond just coding and deploying. You must monitor performance, trace transactions, collect logs, gather metrics, and trigger alarms when metrics breach thresholds. To achieve this, you can design and implement telemetry to enable observability capabilities.

This post is the first in a series of three posts in which I will demonstrate how to instrument a .NET application to generate metrics, logs, and distributed traces and how to use them with Amazon CloudWatch and AWS X-Ray to improve observability. In this first post, I will explore implementing and using Amazon CloudWatch metrics on a containerized .NET workload running in AWS.

Solution overview

Imagine a containerized .NET workload like the one in Figure 1, which runs on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. It comprises three .NET microservices: an ASP.NET Core web API, two Worker Services, and the cloud resources they use. If something goes wrong with this workload, you may see increased error rates, degraded performance, or data inconsistency. How would you troubleshoot the root cause of the problem, such as which components are failing or underperforming?

In this example, the Worker Services have no user interface or API for end-user interaction, so they may fail silently. To avoid such scenarios, you can implement observability using metrics, logging, and distributed traces to provide visibility into the health and performance of each component. Using this architecture as an example, I will show you how to use Amazon CloudWatch and AWS X-Ray to implement metrics in this .NET application.

Architecture Diagram

Figure 1: .NET Microservice solution

Prerequisites

For this example, you can use the GitHub repository “microservices-dotnet-aws-cdk,” which contains the sample code for all three microservices. It uses the AWS Cloud Development Kit (CDK) to define and provision the Infrastructure as code using the C# programming language.

The following prerequisites are required on your system to test and deploy this solution:

You can clone the repository from the command line with the following command:

git clone https://github.com/aws-samples/microservices-dotnet-aws-cdk.git

Open the solution in an IDE like Visual Studio Code to explore the implementation.

Implementing metrics with CloudWatch Metrics in .NET applications

You can implement different metrics in your .NET applications using Amazon CloudWatch. You can collect business-specific metrics that include measurements at the application level, .NET web application metrics to measure HTTP request-related metrics, or infrastructure metrics to provide AWS resource utilization data.

Consider collecting business-specific metrics and .NET web application metrics in the application code. In addition, you can collect infrastructure metrics by using Amazon CloudWatch integration with the AWS compute services where you choose to host the .NET application, such as CloudWatch Container Insights for container-based applications.

To implement the business-specific and .NET web application metrics in the application code, you can use the CloudWatch PutMetricData API call using the AWS SDK for .NET or the CloudWatch Embedded Metric Format (EMF). For this example, I’m focusing on using the CloudWatch EMF to show how it allows the ingestion of complex, high-cardinality application data as logs to generate actionable metrics. EMF enables the application to embed custom metrics in detailed log event data, using a sidecar container to process, parse, and transmit the metrics to CloudWatch. In addition, a sidecar process avoids resource utilization contention in the application, which helps avoid performance and throughput impact.

For the CloudWatch EMF implementation, I am using Amazon.CloudWatch.EMF NuGet package in all three applications projects. I also use the NuGet package (Amazon.CloudWatch.EMF.Web) in the web application project, which allows automatic instrumentation for metrics of all HTTP requests and responses. The following code example illustrates configuring CloudWatch EMF in the Program.cs file to generate metrics for the web application. Refer to one of the Program.cs file of the Worker Service to learn how to implement it for that type of application.

...
//Register CloudWatch EMF for ASP.NET Core
EMF.Config.EnvironmentConfigurationProvider.Config = new EMF.Config.Configuration
{
    ServiceName = MY_SERVICE_NAME,
    ServiceType = "WebApi",
    LogGroupName = Environment.GetEnvironmentVariable("AWS_EMF_LOG_GROUP_NAME"),
    EnvironmentOverride = EMF.Environment.Environments.ECS
};
builder.Services.AddEmf()
...
//Register CloudWatch EMF Middleware
app.UseEmfMiddleware((context, logger) =>
{
    if (logger == null)
    {
        return Task.CompletedTask;
    }
    logger.PutMetadata("MY_SERVICES_INSTANCE", Environment.GetEnvironmentVariable("MY_SERVICES_INSTANCE"));
    return Task.CompletedTask;
});
...

For business-specific metrics implementation, I use dependency injection to supply an instance of an object implementing the IMetricsLogger interface into each controller or service class and emit the business-specific metrics. The following code sample illustrates the implementation.

Injection of the IMetricsLogger instance:

...
 public BooksController(IAmazonSimpleNotificationService client, ILogger<BooksController> logger, IMetricsLogger metrics)
    {
        ...
        _metrics = metrics;
    }
...

Emit business-specific metrics from the application code:

private void EmitMetrics(Book book, string traceId, long processingTimeMilliseconds)
{
    //Add Dimensions
    var dimensionSet = new DimensionSet();
    //Unique Id for this WebAPI Instance
    dimensionSet.AddDimension("WebApiInstanceId", Environment.GetEnvironmentVariable("MY_SERVICES_INSTANCE"));
    //Book's Authors
    dimensionSet.AddDimension("Authors", string.Join(",", book.BookAuthors));
    //Book's Year
    dimensionSet.AddDimension("Year", $"{book.Year}");
    _metrics.SetDimensions(dimensionSet);

    _metrics.PutMetric("PublishedMessageCount", 1, Unit.COUNT);
    _metrics.PutMetric("ProcessingTime", processingTimeMilliseconds, Unit.MILLISECONDS);

    //Add some properties
    _metrics.PutProperty("TraceId", traceId);

    _logger.LogInformation("Flushing");
    _metrics.Flush();
}

Provisioning the CloudWatch agent to transmit the metrics

You must install and configure the CloudWatch agent to ingest, parse, and transmit EMF logs to Amazon CloudWatch. The configuration of the CloudWatch agent varies depending on how you host your .NET application on AWS. For information about the different hosting and configuration options for the CloudWatch agent, read Using the CloudWatch agent to send embedded metric format logs. I am using Amazon ECS and AWS Fargate for this sample application. The following code example shows implementing a CloudWatch agent sidecar for an Amazon ECS service using AWS Cloud Development Kit (AWS CDK) to deploy Infrastructure as code. AWS CDK lets you build applications in the cloud, with the benefit of the expressive power of programming languages, such as C#, used here.

...
public static TaskDefinition AddCloudWatchAgent(this TaskDefinition taskDefinition, CloudWatchAgentProps agentProps)
{
    ...
    taskDefinition
        .AddContainer("cwagent", new ContainerDefinitionOptions
        {
            ...
            PortMappings = new PortMapping[]{
            new PortMapping{
                ContainerPort = 25888,
                Protocol = Amazon.CDK.AWS.ECS.Protocol.TCP
            },
            new PortMapping{
                ContainerPort = 25888,
                Protocol = Amazon.CDK.AWS.ECS.Protocol.UDP
            }},
            Image = ContainerImage.FromRegistry("public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest"),
            Environment = new Dictionary()
                {
                    { "CW_CONFIG_CONTENT", "{\"logs\":{\"metrics_collected\":{\"emf\":{}}}}" }
                },
            ...
        });

     //Grant permission to the cw agent 
    taskDefinition.TaskRole
        .AddManagedPolicy(ManagedPolicy.FromAwsManagedPolicyName("CloudWatchAgentServerPolicy"));

     return taskDefinition;
}
...

Deploy the example solution

The repository contains the full implementation of this solution, allowing you to make an HTTP request to the sample Web API to test it. To deploy, run one of the following deployment scripts in your environment using either bash (Linux or Mac) or PowerShell to deploy the solution.

Using bash:

./deploy.sh

Using PowerShell:

.\deploy.ps1

After deploying, copy the URL printed by the deployment script. It has the following format: http://WebAp-demos-XXXXXXXX-99999999.us-west-2.elb.amazonaws.com/api/Books. The X and 9 should be alphanumeric characters representing your deployment’s unique ID. Then, using a REST API client (such as Thunder Client for VS Code), test the solution by submitting an HTTP POST request to the URL with the following JSON payload. When you submit the HTTP POST, you should receive status 200 and a response indicating the TraceId result.

{
    "Year" : 2022,
    "Title": "Demo book payload",
    "ISBN": 12345612,
    "Authors": ["Author1", "Author2"],
    "CoverPage": "picture1.jpg"
}

Figure 2 illustrates the example API call.

Example API call

Figure 2: Example API call

Visualizing the results

After making a test request to the API, you can navigate to Amazon CloudWatch to review the generated metrics.

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose Metrics, All metrics.
  3. On the Browse tab, the console displays three types of metrics in the Custom namespaces section (Figure 3).

Custom Metrics

Figure 3: Custom namespace metrics

The three types of metrics collected in this solution are:

  1. ECS/ContainerInsights: Captures the Cloud Infrastructure metrics, which are Amazon ECS container-related metrics. These are available because I have enabled CloudWatch Container Insights.
  2. aws-embedded-metrics: Captures the “business-specific” metrics, which are the custom metrics implemented in .NET application code to emit data related to business logic.
  3. demo-web-api: Captures the NET Core web application metrics. These are auto-instrumented for all HTTP requests and responses from the web API microservices.

ECS/ContainerInsights: Cloud Infrastructure metrics from CloudWatch Container Insights

These are performance and health metrics for Infrastructure, displaying information such as CPU utilization, memory utilization, networking, and other key performance indicators for the Amazon ECS workload. You can use them to generate charts, dashboards, alarms, or troubleshoot issues. For example, if the CPU or memory utilization spikes, it could indicate an anomaly. You can define thresholds to alarm in such circumstances. Amazon CloudWatch provides an out-of-the-box performance dashboard for Container Insights metrics. To visualize this dashboard like the one in Figure 4, follow these steps:

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose Container Insights in the Insights
  3. Change the option from Resources to Performance monitoring in the dropdown list on the page.

Sample Container Insights dashboard

Figure 4: Sample Container Insights dashboard

aws-embedded-metrics: Business-Specific metrics

Figure 5 is an example of “Business-specific” metrics visualization. For example, in the Worker Service microservices, the application tracks custom metrics defined as “ProcessingTime,” which captures the processing time of one piece of business logic. CloudWatch allows you to use custom metrics to generate charts, dashboards, or alarms. You can also use the metrics to troubleshoot the time each Worker Services took to complete the processing. To visualize this Metric in the example application, follow these steps:

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose All metrics in the Metrics
  3. On the Browse tab, select aws-embedded-metrics and choose the dimension The console should list the WorkerId instances and the Metric name.
  4. Choose two different WorkerId instances for the same “ProcessingTime”
  5. Change the dropdown list on the top of the page from Line to
  6. Choose the Options Under the Gauge range, set values for Min and Max.
  7. Set the values for percentages as Min to 0 and Max to 100.

aws-embedded metrics dashboard

Figure 5: aws-embedded metrics dashboard

demo-web-api: .NET Web application auto-instrumented metrics

Amazon CloudWatch also allows the same capabilities for framework-related metrics. In this example, CloudWatch groups the automatically instrumented metrics for the ASP.NET 6.0 web API in the namespace “demo-web-API.” You can use the metrics related to HTTP requests/responses to build charts, dashboards, and alarms or to troubleshoot the response time of each endpoint controller action.

To visualize a Metric chart like the one in Figure 6, follow these steps:

  1. In the AWS console, navigate to Amazon CloudWatch.
  2. In the navigation pane, choose All metrics in the Metrics
  3. On the Browse tab, select demo-web-api and choose the dimension “Action, Controller, StatusCode.”
  4. Choose the two options from the list for the same Metric Name of “Time.”
  5. Change the dropdown list from Line to Number at the top of the page.

ASP.NET automatically instrumented methods

Figure 6: ASP.NET automatically instrumented methods

Cleanup

To avoid unexpected charges, you should clean up the resources created by running this demo. To do so, run the script from the root folder where you’ve cloned the GitHub repository.

Using bash:

./clean.sh

Using PowerShell:

.\clean.ps1

Conclusion

In the first part of this blog post series on implementing observability for your .NET applications on AWS, I have covered how to instrument your modern .NET applications to generate CloudWatch Embedded Metric Format (EMF) data points. I have also shown how CloudWatch uses these data points from your application to simplify troubleshooting or visualize your application’s health. To learn about Logs or Distributed Tracing implementations, read the other two parts of this series, “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 2 – Logging” and “.NET Observability with Amazon CloudWatch and AWS X-Ray: Part 3 – Distributed Trace”.

Observability goes beyond instrumenting your .NET application to generate traces, logs, and metrics. You can use Amazon CloudWatch capabilities for monitoring, alarms, or detecting anomalies. Refer to the Amazon CloudWatch Features page to learn more. For a more hands-on experience, check out the One Observability Workshop.

AWS can help you assess how your company can get the most out of cloud. Join the millions of AWS customers that trust us to migrate and modernize their most important applications in the cloud. To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS. Contact us to start your migration journey today.

TAGS: , ,
Ulili Nhaga

Ulili Nhaga

Ulili Nhaga is a Cloud Application Architect at Amazon Web Services in San Diego, California. He helps customers migrate, modernize, architect, and build highly scalable cloud-native applications on AWS. Outside of work, Ulili loves playing soccer, running, cycling, Brazilian BBQ, and enjoying time on the beach.