Rapid7 Reduced Operational Latency by 40% by adopting Amazon DynamoDB

Built on the self-managed Apache Cassandra in 2013, Rapid7’s InsightIDR infrastructure was comprised of 6 clusters. The biggest cluster “Process Registry” had grown to over 280 nodes over a period of 9 years. Their critical application was running on a Cassandra cluster with 80 nodes and was due for a database upgrade. The cluster was hosting multiple-tenants and also co-hosted multiple applications. Any need for patching, version upgrade, or maintenance amplified the impact when it came to managing the cluster. Guru Bandari, Rapid7’s Senior Software Engineer said, “Scaling the Cassandra clusters and performing maintenance tasks such as patching had become an operational nightmare especially with the growing customer base. Scheduling downtime while increasing tenants had become even more challenging than before.”

Rapid7’s use case also included the need to delete data after 60 days of its creation on the InsightDR application. Previously, they relied on Cassandra's TTL (Time to Live) feature to remove the data, but this approach consumed resources and adversely affected performance.

As the popularity of InsightIDR product increased, Rapid7 was looking for a maintenance free, fully managed, and highly available database solution.

In June 2022, Rapid7 modernized their Cassandra workload by migrating to Amazon DynamoDB. Rapid7 technical team saw value in DynamoDB, primarily because it is a fully managed database service with up to 99.999% availability. They worked on redesigning their application to adopt DynamoDB which included endurance testing with peak workloads. They benefited from DynamoDB’s seamless scaling and low single-digit milliseconds latency for their read and write workloads.

For a critical and multi-tenant application such as InsightDR, Rapid7 adopted the safest migration approach of enabling dual writes at the unpredictable application layer, using custom scripts. To address this challenge, they utilized DynamoDB's On-Demand mode, enabling them to effectively manage unexpected surges in traffic and analyze traffic patterns. Rapid7 continued to run reports and test their read-workloads on dual environments until they confirmed data consistency.

They also used Amazon CloudWatch Contributor Insights, which was crucial for identifying hot partition issues, which was identified as bottle-neck during endurance testing. After running their application workload in parallel for two months across both environments, they gained the confidence to modernize their entire workload with DynamoDB which helped them achieve the near zero downtime migration.

Rapid7 was also drawn to Amazon DynamoDB due to its integrated security features and native support for backup and restore. Additionally, the point-in-time recovery (PITR) feature of Amazon DynamoDB proved highly beneficial as it allows Rapid7’s customers to effortlessly and continuously back up their table data with precise per-second granularity. This feature enabled Rapid7 with the ability to restore their data to any specific moment within the past 35 days.

DynamoDB's TTL feature provided automated deletes to reduce stored data volumes without any additional cost or performance impact. Andrew Keely, Senior Manager Software Engineering shared, “Rapid7 experienced improvements in their annual infrastructure management costs and reduced an average of 60 weekly hours by modernizing with Amazon DynamoDB. Moreover, this migration enabled them to achieve consistent performance with single-digit milliseconds latency and ensured operational reliability with 99.999% availability for their customers.”