Automate time series network visualizations for AWS PrivateLink using Amazon CloudWatch Contributor Insights

AWS PrivateLink is a highly available, scalable technology that lets you connect your Amazon Virtual Private Cloud (VPC) to supported AWS services without requiring public internet traversal. It also lets you privately connect to services hosted by other AWS accounts (VPC endpoint services) and supported AWS Marketplace partner services.

Amazon CloudWatch Contributor Insights is a feature of Amazon CloudWatch that you can use to analyze log data to create time series that display contributor data. This will help you understand who or what is impacting your system and application performance. It does this by identifying top talkers, pinpointing outliers, finding the heaviest traffic patterns, and ranking the top system processes.

In this post, I describe a solution I built to provide 1-click automation to set up time-series monitoring and visualization for PrivateLink connections using Contributor Insights and AWS CloudFormation. The solution provides full automation in CloudFormation to create custom Amazon CloudWatch dashboards, Amazon CloudWatch metrics, and Amazon CloudWatch alarms based on Contributor Insights rules. These rules can monitor several aspects of PrivateLink connections, such as bytes transferred by source and destination IP address, traffic by source address and action or top sources for rejected TCP connections. You can download and use the solution for visualizing both the consumer side or the provider side of the PrivateLink connection.

Solution architecture

How AWS PrivateLink works

AWS PrivateLink works by creating a VPC endpoint for the supported service in your VPC. This supported service can be a supported, native AWS service, or it can be your own application inside your VPC. If the service is your application inside of a VPC, then it’s called a VPC Endpoint service.  In that case, other AWS principals create VPC endpoints from their VPC to your endpoint service.

With PrivateLink, the service owner is the service provider, and the principal creating the VPC endpoint is the service consumer. In both cases (AWS service or VPC Endpoint service), on the consumer side, PrivateLink works by using a VPC endpoint that creates Amazon Elastic network interfaces in your consumer VPC subnets. Each elastic network interface provides a private IP address that serves as an entry point for traffic destined to the provider service. On the provider side, for a VPC Endpoint service where you have your own application to be shared, PrivateLink requires the deployment of a Network Load Balancer or a Gateway Load Balancer as a VPC Endpoint service. Then, you enable the Load Balancer to be shared with other consumer VPCs. When you allow sharing with consumer VPCs, you can also require explicit acknowledgement from the provider side for consumer accounts to create endpoints in their VPC for your endpoint service.

Solution overview

1. Configure flow logs from consumer and provider VPC and publish data to Amazon CloudWatch Logs. In this solution, you first configure a VPC Flow Log. A VPC Flow Log is a feature that lets you capture information about IP traffic going to and from network interfaces in your VPC. You configure a VPC Flow Log for the consumer VPC where your VPC endpoint is deployed. This VPC Flow Log captures IP traffic associated with each elastic network interface from each consumer subnets where the VPC endpoint is deployed. In the case of a VPC endpoint service, you configure a VPC Flow Log for the VPC where the Network Load Balancer is deployed. In this case, there is also one elastic network interface per load balancer subnet.
   1. You configure the flow logs (from both the consumer VPC in the case of a VPC endpoint as well as the provider VPC in the case of a VPC Endpoint service) to publish flow log data directly to Amazon CloudWatch Logs. When publishing to CloudWatch Logs, flow log data is published to a log group, and each elastic network interface has a unique log stream in the log group. These log streams contain flow log records that capture IP traffic going to and from that individual elastic network interface.
2. Automate provisioning of Contributor Insights rules My solution automates the provisioning of Contributor Insights rules that are applied to CloudWatch Logs. These rules filter log data based on log streams that correspond to individual elastic network network interfaces in consumer VPC endpoint subnets or provider Network Load Balancer subnets. I use the INSIGHT_RULE_METRIC metric math function to automate the provisioning of CloudWatch metrics from these rules, and then provision CloudWatch Alarms based on thresholds applied to these metrics.
3. Automate provisioning of a CloudWatch dashboard Finally, the solution automates the provisioning of a CloudWatch dashboard that displays time series visualizations of network traffic in your PrivateLink based on these Insights rules.

Prerequisites

1. Set up a PrivateLink connection. In this post, I’ve deployed a PrivateLink connection from our AWS Networking Workshop Immersion Day available here. The Intermediate Topics section of the Networking Immersion Day guides you through step-by-step instructions to set up an end-to-end PrivateLink that demonstrates both the consumer-side setup using a VPC endpoint as well as a provider-side setup using a VPC endpoint service.
   1. Complete Lab #5. As mentioned in the Lab inter-dependency section of the Networking Immersion Day, completing Lab#5 requires the following:
      1. Lab #1 deployment and completion
      2. Lab #2 deployment and completion
      3. Lab #3 deployment
   2. Note the identifiers of your elastic network interfaces that were provisioned for the VPC endpoint. Navigate to the VPC console. On the left panel, select Endpoints. In the console the Service name starts with, and search for an Endpoint ID ‘amazonaws.vpce.regionname.vpce-svc-‘. This the endpoint that is created in your consumer VPC (NP1 VPC) for the Endpoint Service that is deployed in the provider VPC (NP2 VPC). Select the Subnets tab, and note the Network Interface ID (starts with ‘eni-‘) for the elastic network interface in each of the two subnets where this endpoint is deployed in the NP1 VPC.
   3. Note the identifiers of your elastic network interfaces that were provisioned for the Network Load Balancer used by the VPC Endpoint service. Navigate to the EC2 console. Select Load Balancers, and you should see a Network Load Balancer named ‘tgw-NP2-NLB’ that has been provisioned for you. The DNS name for this Network Load Balancer should be ‘tgw-NP2-NLB-<identifier>.elb.<region>.amazonaws.com’. Note the identifier value from the DNS name. From the left panel of the EC2 console, select Network Interfaces, and in the console search for Network interfaces where the Description contains ‘ELB net/tgw-NP2-NLB/<identifier>’. Note the Network Interface ID (starts with ‘eni-‘) for the elastic network interface in each of the subnets where this Network Load Balancer is deployed in the NP2 VPC.
2. To set up VPC Flow Logs and publish flow logs to CloudWatch Logs, follow the step-by-step instructions. Do this for both the NP1 VPC and the NP2 VPC. Note the name of the CloudWatch Log group for the VPC Flow Log for the NP1 VPC, as well as for the NP2 VPC.

Solution setup: Automate time series network visualizations for AWS PrivateLink using Amazon CloudWatch Contributor Insights

The AWS CloudFormation template and a detailed README for this solution are available here. The solution is installed in a single step:

1. Create a stack from the AWS CloudFormation console by launching the aws-privatelink-cloudwatchcontributorinsights.yml
   1. The template takes the following as input parameters:
      1. Network Interface ID of any one of the elastic network interfaces for the VPC endpoint. This is in Step 1b in the prerequisites section).
      2. Network Interface ID of any one of the elastic network interfaces for the Network Load Balancer used by the VPC Endpoint service. This is Step 1c in the prerequisites section.
      3. Name of the CloudWatch Log group configured for the VPC Flow Logs for the consumer VPC. This is NP1 VPC from Step 2 in the prerequisites section.
      4. Name of the CloudWatch Log group for the VPC Flow Log for the provider VPC. This is NP2 VPC from Step 2 in the prerequisites section.
      5. Email address where you can receive notifications and any name for the Amazon Simple Notification Service (Amazon SNS) topic used by CloudWatch Alarms.

Test and validate the Contributor Insights PrivateLink solution

To test the PrivateLink, you must access the provider’s VPC endpoint service from the consumer’s VPC endpoint.

In this post, I use the setup from the Networking Immersion day. I access the target web application in the provider VPC (NP2 VPC) from the VPC Endpoint in the consumer VPC (NP1 VPC) as shown in the following diagram:

1. Set up DNS alias

I’ve already created an Amazon Route53 DNS entry for the VPC Endpoint here based on the lab instructions (Lab #5). My DNS record web.np1.aws. example.com is associated with an alias to the DNS name created for the endpoint, which starts with vpce and ends with amazonaws.com.

2. Generate traffic
   1. Navigate to the EC2 console and select the ‘NP1-tgw-Server’.
   2. From the top panel, select Connect. To access the instance, from the Session Manager tab on the next screen, select Connect.
   3. To generate traffic and hit the target web applications from your VPC Endpoint every two seconds, enter watch -n 2 "curl web.np1.aws.example.com -s".
   4. In the response from the target web applications, you will see a message indicating their private IP address.
3. Access your Contributor Insights PrivateLink dashboard in CloudWatch
   1. Navigate to the CloudWatch console.
   2. From the left panel, select Dashboards, and select the ‘PrivateLinkContributorInsights’ dashboard that has been provisioned by the solution.
   3. You will now see real time, time-series visualizations for the total number of bytes transferred or the total number of connections for your PrivateLink connection. The following screenshot shows my PrivateLink Total dashboard, one chart with Total Connections and one with Bytes Transferred.

4. View your Cloudwatch metrics and alarms
   1. From the CloudWatch console, select All alarms.
   2. In the filter panel, search using the ‘PrivateLink’ You will see two CloudWatch Alarms, PrivateLinkHighBytesTransferAlarm and PrivateLinkRejectConnectionsAlarm.
   3. To view the alarm that was triggered when the configured threshold for transferred bytes within an interval period was breached, select the PrivateLinkHighBytesTransferAlarm.
   4. To view the High_Bytes_Transfer metric provisioned by the solution for the PrivateLink connection using the INSIGHT_RULE_METRIC metric, select View in metrics.

The following screenshot shows the chart for PrivateLink high bytes transfer alarm.

Cleanup

To clean up your account after deploying the solution outlined in this post, delete the CloudFormation stacks for the aws-privatelink-cloudwatchcontributorinsights.yml template. Delete each of the templates deployed for the Networking Immersion Day in the reverse deployment order: Labs #5, #3, #2, and #1.

Conclusion

In this post, I presented a solution that provides 1-click automation to set up time series-based monitoring and visualization for PrivateLink connections using Contributor Insights and CloudFormation. Download and use the solution both on the consumer side or the provider side of the PrivateLink connection.

About the author