Calculate Amazon DynamoDB reserved capacity recommendations to optimize costs

This post is co-written with Sanjna Srivatsa, Data Scientist at VMware.

Amazon DynamoDB provides discounts ranging from 50 percent to 77 percent to customers who commit to paying for a minimum usage level, called reserved capacity. In this post, you learn how to use historical DynamoDB usage data to make your first DynamoDB reserved capacity commitment.

As a FinOps practitioner (and data scientist), my focus is to optimize spending on the cloud technologies that help power VMware. At VMware, we used a considerable amount of provisioned capacity for DynamoDB, but we struggled to find the right method to make informed reserved capacity purchases. Because no tools or tutorials were available, I developed my own method that I want to share with others.

DynamoDB capacity and billing modes

First, it’s important to understand the different DynamoDB capacity and billing modes. DynamoDB offers two modes for managing table read and write capacity: on-demand and provisioned. On-demand mode automatically adapts to workload changes, and you’re billed for each read and write request. Provisioned capacity mode requires you to specify the number of reads and writes a table can support per second, and you’re billed for the number of provisioned reads and writes per hour. Provisioned capacity mode is less expensive than the equivalent on-demand mode, and is a good choice for workloads with predictable traffic.

Provisioned write capacity is provisioned in write capacity units, or WCUs. Each WCU provides the ability to write up to 1 KB in size per second. If you need to write 100 KB per second to your table, you would provision at least 100 WCUs. Provisioned read capacity is provisioned in read capacity units, or RCUs. Each RCU provides the ability to perform one strongly-consistent read up to 4 KB in size per second or two eventually consistent reads per second to read up to 8 KB in size per second. If you need to read 400 KB of strongly consistent data per second, you would provision at least 100 RCUs or 50 RCUs of eventual consistency reads.

DynamoDB reserved capacity

DynamoDB offers the option to purchase reserved capacity for provisioned capacity mode (on-demand isn’t currently supported for reserved capacity). With a 1- or 3-year commitment, reserved capacity provides a significant discount on the cost of provisioned capacity. Note that the 3-year commitment is only available in some AWS Regions.

When you purchase reserved capacity, you specify the following:

Region in which the reserved capacity will apply
Read or write capacity (RCUs or WCUs)
The number of capacity units (reserved capacity is purchased in increments of 100)
The term (1 or 3 years)

A capacity reservation discount will apply to all WCUs and RCUs in the specified Region, up to the number of capacity units purchased. To minimize cost, your usage is first compared to the reserved capacity purchased. Each hour, if the amount of capacity used is less than or equal to the total reserved capacity you have purchased, all capacity will be charged at the reserved capacity rate. Any excess throughput is billed at the standard provision capacity rates. For example, if you buy 100 WCUs and your actual usage is 100 WCUs or less, you will be charged the reserved capacity rate. Any capacity that you provision in excess of your reserved capacity (above 100 WCUs in this example) is billed at standard provisioned capacity rates.

Note, if you use AWS Organizations and have multiple accounts linked with consolidated billing, reserved capacity units purchased either at the payer account level or linked account level are shared with all accounts connected to the payer account. Reserved capacity will first be applied to the account that purchased it, and then any unused capacity will be applied to other linked accounts.

In addition, reserved capacity is not currently available for replicated WCUs for global tables. Reserved capacity is also not currently available for tables using the DynamoDB Standard-IA table class or on-demand capacity mode.

The following section describes how you can calculate a conservative reserved capacity purchase.

Calculate how much reserved capacity to purchase

The AWS Cost and Usage Report (CUR) is a detailed report that breaks down AWS costs across multiple dimensions. You can schedule the CUR to deliver to an Amazon Simple Storage Service (Amazon S3) bucket of your choice. If you haven’t enabled the CUR, see Creating Cost and Usage Reports.

Before we go further, it’s important to pause for a moment and think about the seasonality of your application. Just as your data might have seasonality throughout the year, it can have daily seasonality. Perhaps your traffic is heaviest in the morning hours or in the afternoon. VMware uses daily data to calculate the reserved capacity purchase amount. However, we recommend that you use hourly data to start so that you can better understand the daily fluctuations of your data. Furthermore, it’s suggested to look back at least 30 days to establish a good period of time to consider. But be mindful of yearly seasonality when you choose a 30-day lookback period.

Next, you’ll extract data and chart it to find the initial reserved capacity purchase amount.

Extract relevant data from the CUR

You can access and query the CUR with Amazon Athena, or with third-party tools and pipelines. Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage and you pay only for the queries that you run. You can select the tool of your choice to run the following query, which extracts the necessary data to calculate your first purchase. Note that running these Athena queries might incur costs (you can check the Athena pricing page for more information).

The following query extracts data at an hourly granularity for the previous 30 days:

SELECT
DATE_TRUNC('hour', line_item_usage_start_date) AS line_item_usage_start_date,
	line_item_line_item_type,
	product_group,
	product_region,
	line_item_product_code,
CAST(SUM(reservation_unused_quantity) AS DECIMAL(38,3)) AS reservation_unused_quantity,
CAST(SUM(line_item_usage_amount) AS DECIMAL(38,3)) AS line_item_usage_amount
FROM 
	<table_name>
WHERE 
	line_item_product_code IN ('AmazonDynamoDB')
	AND line_item_line_item_type = 'Usage'
	AND product_group IN ('DDB-ReadUnits', 'DDB-WriteUnits')
	AND line_item_usage_start_date BETWEEN DATE_ADD('day', -30, CURRENT_DATE) AND CURRENT_DATE
GROUP BY 1,2,3,4,5
ORDER BY 1,2,3;

Note: You will need to replace <table_name> with the Athena table name for the CUR file.

To learn more about querying the CUR, see Querying Cost and Usage Reports using Amazon Athena.

Find the right reserved capacity units to buy

After extracting the data, the next step is to import it into your data analytics tool of choice, such as Amazon QuickSight. Depending on the amount of data, it’s possible to use spreadsheet programs like Microsoft Excel as your analytics tool. After the data is imported, it’s important to format columns with the correct data types so the data can be correctly charted. For example, date columns should be set to the date data type.

Finally, you need to visualize the usage patterns. It’s best to visualize the data in a manner that shows the trends and variability of your usage. It can help you understand if your provisioned capacity usage is growing or reducing, and if the usage varies per hour. To do this, you can use any data visualization tool; as mentioned before, depending on the size of your data, you can use Excel. The following are my recommendations for visualizing the data:

Filter the data by Region (because rates differ by Region)
Chart type: Line
X-axis: line_item_usage_start_date
Granularity: Hourly
Y-axis: line_item_usage_amount (average)
Repeat-by: product_region
Color: product_group

If your DynamoDB use spans multiple Regions, you’ll need to calculate the reserved capacity purchase amount by Region. When you’re done, your chart should look something like Figure 1 that follows.

Figure 1: Chart showing a mock data sample of hourly consumption rates of data read and write capacity units

The column lineitem_usageamount for DynamoDB product lineitems records usage quantity per-second. To normalize usage amount to hourly granularity, this column must be divided by 3,600 (seconds). You may want to perform normalization before creating a visualization. The preceding chart uses simulated data at an hourly granularity. You can create a separate chart for each Region. Note that several hours in the preceding chart have a spike in usage, so you will want to adjust your reserved capacity purchase accordingly. This shows the importance of understanding the hourly usage patterns before purchasing reservations. Large spikes in read/write units over a few hours can skew daily averages, which is why daily or monthly granularity might not work to derive a purchase recommendation for everyone’s usage patterns.

If you’re new to reservations or if this is your first DynamoDB reservation purchase, it is recommended to start small to get comfortable with the process and make iterative purchases. Let’s take the chart above as an example: because there is stable usage in the Region, the first purchase can be about 50 percent of the minimum usage amount (Y-axis) or in this case 110,000 WCUs and 28,000 RCUs. Again, you need to do this same analysis for each Region to determine your reserved capacity purchase by Region. The next post on this topic will help you monitor your purchases, evaluate reserved capacity purchase consumption and make iterative purchases.

Make your first purchase

The final step is to make a reserved capacity purchase with the charted data. To do so, you can navigate to the Reserved capacity page on the AWS Management Console for DynamoDB. It’s important to make sure you’re in the correct Region. Choose whether you want to purchase read or write capacity units, the term, and then enter the amount. Remember that the amount you enter is your hourly commitment. After you’ve filled out the form, choose Purchase reserved capacity.

Conclusion

With the process outlined in this post, you can derive a safe number of reserved capacity units to purchase for each Region. We recommend recreating these charts regularly to understand if there have been any changes in your usage and revising your future purchases accordingly.

Stay tuned for the next post on this topic, which will help you monitor your purchases, make iterative purchases, and deal with some edge scenarios.

About the Authors

Sanjna Srivatsa is a Data Scientist with the Cloud Business Office (CBO) focused on Public Cloud Optimization at VMware. She is interested in solving complex FinOps problems using data science.

Kevin Willis is a Product Manager on the Amazon DynamoDB team. He currently focuses on the programmability area of the product.

Sahil Thapar is an AWS Principal Solutions Architect. He works with ISV customers to help them build highly available, scalable, and resilient applications on the AWS Cloud. He is currently focused on containers and machine learning solutions.

AWS Database Blog