AWS Cloud Operations Blog

Using AWS Cost and Usage Reports and Cost Allocation Tags to understand VPC Flow Logs data ingestion costs in Amazon S3

AWS customers enable the VPC Flow Logs feature in their accounts for security, governance, and auditing. They often have several teams who create VPC flow log subscriptions for their workloads and publish the data  to the same Amazon Simple Storage Service (Amazon S3) bucket as part of a centralized logging architecture. Customers need a way to categorize their bills so they can accurately charge back those internal teams who create VPC flow log records. Customers who are considering the use of VPC flow logs often don’t know what their costs will be. Creating a few flow logs and tracking charges can help customers accurately estimate their costs before they enable the feature across their environment.

With the recent support of cost allocaton tags for VPC flow logs, customers can now tag their VPC flow logs to track charges from publishing flow logs to S3 buckets. To track charges from publishing flow logs to Amazon CloudWatch Logs, customers can apply cost allocation tags to their destination log group.

In this post, we  will provide steps for using cost allocation tags to analyze VPC flow log costs and charge back the responsible business unit.

Solution Overview

The  setup consists of the following configuration steps on the AWS console:

Create a VPC flow log

1. In the Amazon VPC console, choose your VPC.

2. On the Flow logs tab, choose Create flow log with S3 as the destination. Repeat these steps for as many VPCs as you like. To create a flow log and setting the destination to an Amazon S3 bucket you can follow the steps in our documentation.

In the Flow logs list, the sales-all-s3 log is selected. Its filter is set to ALL. Its destination type is S3. Its destination name is vpc-flowlog-aggregate-bucket.

Figure 1: Flow logs tab

Add tags to VPC flow logs

1.     Select the flow log and then from Actions, choose Manage tags.

2.     For chargeback, add tags to indicate the Line of business unit (LBU) that owns this VPC. Keep the tag name consistent across all VPC flow logs. In Figure 2, the tag name is cost-center.

In the screen for Manage tags, a tag named cost-center has a value of sales-bu.

Figure 2: Cost-center tag

Activate user-defined cost allocation tags

1.    Open the Billing and Cost Management console and from the left navigation pane, choose Cost allocation tags.

The left navigation pane displays Cost Management, Cost Explorer, Budgets, Budgets Reports, Savings Plans, Cost and Usage Reports, Cost Categories, and Cost allocation tags.

Figure 3: Billing and Cost Management Dashboard

2.    Under User-defined cost allocation tags, search for your tag name and then activate the tag. This ensures that Cost Explorer and your AWS Cost and Usage Reports will include these tags.

Under User-defined cost allocation tags, cost-center is entered in the search field. It appears in a list with a status of Active.

Figure 4: Cost allocation tags

Analyze VPC flow log costs in the AWS Cost and Usage Report

1.  Follow the steps in Querying Cost and Usage Reports using Amazon Athena.

Query editor in Amazon Athena console displays sample queries and the tables derived from the AWS Cost and Usage Reports.

Figure 5: Query editor in Amazon Athena

2.     Use the following SQL query to analyze the cost of VPC flow logs per cost center.

SELECT 
SUM(line_item_usage_amount) AS total_usage_amount, 
SUM(line_item_unblended_cost) AS total_cost, line_item_product_code, 
line_item_operation, 
product_usagetype, 
line_item_line_item_description, 
resource_tags_user_cost_center
FROM "cur_db"."cur_data" 
WHERE
 line_item_usage_type like '%S3-Egress%' AND 
line_item_product_code like '%AmazonCloudWatch%' AND 
line_item_operation like '%LogDelivery%' 
GROUP BY 
resource_tags_user_cost_center, 
line_item_product_code, 
line_item_operation,
product_usagetype, 
line_item_line_item_description
Query results are organized in a table with columns for total usage amount, total cost, line item product code, line item operation, product usage type, line item description, and resource tags user cost center.

Figure 6: Query results

AWS Cost Explorer view

Customers can also use the filters in AWS Cost Explorer to view VPC flow log charges grouped by cost allocation tags.

The results are displayed in a table where sales-bu ($) and sales-bu (GB) are highlighted. The page also displays Cost and Usage graphs for sales-bu and core-infra-s3.

Figure 7: Group by cost-center tag

Conclusion

In this blog post, I’ve shown how customers can analyze  VPC flow log data ingestion costs using cost allocation tags and AWS Cost and Usage Reports.  This will help customers to chargeback business units for the VPC Flow Logs that they are generate.  While implementing tagging also be sure to follow AWS tagging best practices.

About the authors

Kalyan Janaki is Senior Technical Account Manager with AWS. Kalyan enjoys working with customers and helping them migrate their workloads to the cloud. In his spare time, he tries to keep up with his 3-year-old.

 

 

 

Abhishek Lahiri is Senior Technical Account Manager with AWS. Abhishek helps large enterprises build secure and scalable solutions on the AWS cloud, helping them with cloud transformation initiatives.