How Kurtosys quickly completed a massive cross-Region data migration using Amazon S3 Batch Replication

Kurtosys helps investment managers accurately deliver pain-free data and insights to their clients at scale. From client reporting and investor portals to global investment website solutions, their capabilities are focused on making marketing and client reporting teams’ lives easier. Their secure, cloud-based platform offers a set of easy-to-use tools designed to save clients time and optimize cost while offering secure and compliant results. Underpinned by global ISO-27001 certification in information security management, Kurtosys has a deep understanding of data, information design, industry regulations, and compliance issues. Trusted by some of the world’s largest and leading investment brands, Kurtosys’ global team collaborating every day to help investment managers succeed and grow.

In this blog post, we discuss how Kurtosys migrated over 100 TB of data across AWS Regions, including near real-time replication, using Amazon S3 Batch Replication. We also discuss how this enabled Kurtosys to achieve data residency and data isolation requirements mandated by their customers.

Kurtosys’ use case

As part of Kurtosys’ digital modernization and innovation in November 2021, among the several objectives of modernization, data migration became one of the most important tracks, and stakeholders stressed the following key aspects:

• Guarantee that data existed as a dynamic mirror in two locations (source and destination until the migration is fully done. This would make sure that Kurtosys could decommission the source Amazon Simple Storage Service (Amazon S3) buckets and subsequently the destination S3 buckets would assume full data responsibilities.
• Meet the data residency and data isolation requirements mandated by some existing customers and the same can be easily followed in future.
• Making sure that security best practices (security is the top priority at AWS) are not compromised.
• How data would be stored and moved while mitigating outages and executing efficiently against a time sensitive roadmap.
• How to remain agile in the face of unforeseen challenges and circumstances during implementation and meeting the larger program timeline.
• The solution must be cost-effective.

The main objective of the data migration was to ensure that new and existing data (S3 objects) from numerous origin S3 buckets in a single AWS account in Region A can be moved to the corresponding target S3 buckets across multiple AWS accounts in Region B. The data move was required mainly to comply with the data residency requirement by some of Kurtosys’ customers. There was also a requirement by some customers to move their data to an isolated AWS account. The challenge offered an opportunity to dive deep into several AWS service offerings and avoid any 3^rd party tool (which would otherwise introduce additional cost and delays) for the migration.

Amount of data to be migrated across Region

• Over 100 TB of data across multiple S3 buckets
• Over 600 million objects

Constraints and known factors at Kurtosys

• New objects are being added 24/7 in the source bucket, which must be replicated immediately (near real-time) to the destination bucket.
• All existing data in the source must be copied to the destination for all buckets.
• Data to be copied across Region and cross accounts. However, there was an option to delete very old data before migration using lifecycle policies, which was comprised of only 5% of the total data to be migrated.
• Solution must be cost efficient and should be able to migrate the existing data within days.

Solution approach

While the team was conducting the POC using existing available AWS Services, AWS announced the Amazon S3 Batch replication capability (a combination of Amazon S3 Batch Operations and Amazon S3 Replication) on the 8th of February 2022. Given that it’s a native service within Amazon S3, the Kurtosys team wanted to immediately explore this new all-in-one feature.

The key terms here in the S3 Batch replication announcement were “existing objects” and “any amount of data.” These were the exact parameters needed in a solution to solve the current list of concerns and challenges. Although it was exciting that a theoretical solution was found, the team needed to engage in a practical implementation to be convinced.

A great point worth mentioning is that S3 Batch Replication works in combination with existing Amazon S3 Cross-Region Replication (CRR). This was convenient for the current setup, since there were already ‘replication’ rules configured on the specified source buckets. The existing Amazon S3 CRR rules make sure that new objects in the source buckets are copied to the destination.

Figure 1: S3 Batch Replication and Amazon S3 CRR working together

Now we had to address the job of replicating the existing millions of objects from the source buckets to the destination buckets in a different Region. The S3 Batch Replication job is a feature that enables the replication of existing objects in bulk between S3 buckets and works in a simple four step process as outlined in the following.

Step 1: First, the Kurtosys team configures a replication rule on the source bucket and specifies the destination bucket. This rule can be configured to limit the scope by using one or more filters for the object key prefix and tags. or the rule can apply to all objects in the bucket. This replication rule makes sure that all new objects get replicated from the source bucket to the destination bucket.

Step 2: The S3 Batch Replication creates the job and automatically generates a Batch Operations Manifest file (manifest.json) using the existing Amazon S3 replication configuration of the source batch. Creation of manifest file (Amazon S3 scans all objects in the source bucket to list objects eligible for replication based on the enabled replication rules on the source bucket) is an important step in the S3 Batch Replication job setup. It is a snapshot of all prefixes and objects that exist in a bucket at a given time. Furthermore, while creating the replication job optionally specified additional filters based on the ‘ObjectReplicationStatuses’ value can also be applied. It is a JSON file that lists the objects to be replicated and includes important information, such as the source and destination buckets, object keys, and other replication rules. A sample manifest.json file is shown here:

Step 3: All objects captured in the manifest are replicated to the destination bucket with S3 Batch Replication. In the meantime, any new writes to the source bucket during the manifest generation or replication period also automatically get replicated to the destination bucket with live replication (CRR).

Once the manifest compilation is complete, the batch replication process begins transferring the prefixes and objects listed in the manifest file.

Testing the process was successful with a fair amount of data. However, the team at Kurtosys were unsure of how this solution would behave with the large amount of data they needed to transfer (especially after earlier data sync attempts failed).

Eventually, for the use case (as explained earlier) with over 100 TB of data, this solution worked as expected on Kurtosys’ European and United States large buckets, taking approximately 3-4 days. All objects replicated without errors. Full visibility into the job’s progress was available, including the running time and percentage of objects completed in the S3 Batch Operations Job Status page on the Amazon S3 Console.

Figure 2: The output of the successful Batch Replication job in one of the United States Regions (source Region), successfully transferring ~67 million objects in 3-4 days (note that the actual duration to complete the replication would vary based on several factors, such as network connectivity, individual object size, any rules)

The Kurtosys team was satisfied with the outcome of this POC and went ahead with the full migration. The migration of all data completed successfully within the original timeline of the program without any further surprises.

This also addressed one of the major requirements of having a low-cost solution. Kurtosys completed the migration within the budget. The cost of S3 Batch Replication and CRR are detailed here. Note that the overall migration cost included Data Transfer Costs (DTO) for moving the data across the Regions.

Learnings

The key was to explore all possibilities offered by AWS and engage the AWS Service Engineering team while doing large migrations. A close collaboration with AWS Support team made sure that the right and most cost-effective solution was in place.

Apart from collaboration, start early with a POC to make sure that all technical boxes are ticked to eliminate any late surprises. However, POC must be done with a large section of the data to get a real view of what would be the impact of different unknowns (such as network bandwidth, individual object size) before doing production cut over.

Conclusion

In this post, we discussed how AWS helped Kurtosys deploy a highly scalable, reliable solution that helped Kurtosys transition and consolidate 100s of millions of S3 objects across Regions and accounts in a cost-effective way using Amazon S3 Batch Replication. All of this was completed using native Amazon S3 features without involving any additional services. This kept the solution simpler, and the team didn’t have to worry about managing multiple services. This also helps Kurtosys migrate data to the Region where the customer is and achieve data residency requirements mandated by some customers. Also addressed the data isolation requirements from some of the customers where they wanted to move their data to a fully isolated dedicated AWS account.

The successful implementation of this solution, with all presented challenges, was a huge win for Kurtosys in terms of achieving their overall modernization roadmap. It made sure that the other modernization implementations that relied on this data being in the correct location could continue.

As Kurtosys moves even further into the infinite world of technological advancements, these lessons will cultivate a culture of service excellence.

Thanks for reading this post. If you have any questions or suggestions, please leave your feedback in the comments section.