What is Data Replication?

Data replication is the process of copying data from one location to another location, for multiple copies of the same, synchronized data. These copies can reside across different storage systems, databases, or geographic locations. Organizations use data replication for data availability to end users, disaster recovery efforts, and to meet compliance requirements. Data replication strategies range from real-time synchronous replication to scheduled asynchronous replication, with strategies selected based on use case.

Data replication is the process of copying and synchronizing data to multiple locations, called replica servers. Various approaches exist, but the core replication process occurs in five fundamental stages:

Source identification

The data replication process begins by identifying the source system or dataset to replicate. Database tables, file systems, and virtual machines are all common targets. The source identification process also includes identifying which components do not need to be replicated, which can help you to reduce both bandwidth utilization and storage costs.

Change detection

The data source is instrumented to identify changes, so the replicas can update on each change. In databasing, this stage is known as the Change Data Capture (CDC) stage. Here, a data replication service identifies what changes in source data must be copied to replica servers.

You can configure this in several ways:

• Log-based detection: You monitor transaction logs (e.g., database write-ahead logs) to capture every change as it happens. This approach is preferred for databases because it’s efficient and captures changes with low latency.
• Event-driven triggers: An event-driven trigger in each database or system notifies your replication service whenever the database is updated. This approach is often used when you can’t perform log-based detection.
• Timestamp-based detection: Your replication system tracks modifications using file or record timestamps, such as a “Last Modified” column in a database table.

Data transfer

Once changes are identified, the system transfers the relevant data from the source to the target system or systems. This transfer can occur synchronously for mission-critical data or asynchronously, in batches, when you want improved performance and have some degree of risk tolerance.

Replication systems will often compress data to optimize bandwidth. They can also encrypt data before it moves from source to replica servers for added security.

Target update

At the destination, the replication system applies the changes to the replica. For example, that could involve inserting, updating, or deleting database records. It could also mean creating new object versions in cloud storage.

Consistency verification

Discrepancies during replication might appear due to network issues, storage corruption, or other errors. At this stage, the replication system verifies that the updated data on the replica matches the primary source.

Comparing data byte-by-byte across a network is rarely an efficient approach, especially at the scale at which many organizations use data replication. For this reason, systems typically generate and compare checksums or cryptographic hashes (such as MD5 or CRC32) for specific data blocks. If the hash on the primary server matches the hash on the replica, the system confirms the data to be identical. If the system detects a discrepancy, it pinpoints the specific range of mismatched data and initiates a new update.

You can perform data replication several different ways, with each method suiting different use cases.

Synchronous replication

In synchronous replication, the same data is written to the source system and all replica systems at the same time. The source system won’t consider the write operation complete until all replica systems confirm the update. Organizations use synchronous replication to help ensure data integrity in mission-critical resources, such as their core database systems. However, it is sensitive to network latency issues, as the write operation only completes when the slowest replica returns an acknowledgement.

Asynchronous replication

Asynchronous replication means data is written to the source system first, and copied to replica servers at some point in time after that. The source system doesn't need any acknowledgement from replicas to proceed. Asynchronous replication occurs with some lag, which might not be acceptable for mission-critical systems. Asynchronous replication is more tolerant of network latency, at the slightly increased risk of data loss during the phased replication process. It’s commonly used to move data across regions for backup and to gather data for analytics.

Semi-synchronous replication

Semi-synchronous replication balances fully synchronous and asynchronous replication. The primary server will copy data to at least one replica and wait for it to acknowledge receipt before confirming a successful write operation, rather than all replicas. This approach provides better assurance against data loss than asynchronous replication, but is much less sensitive to latency issues than a full synchronous mode. It’s practical for use cases where you want a balance between reliability and performance.

Snapshot-based replication

Snapshot replication captures periodic, single point-in-time copies of data and transfers it to one or more replicas. This approach is comparatively straightforward to implement and is useful for initial seeding or static workloads. However, nothing is captured between snapshots, so it’s not suitable for use cases where you need near real-time recovery. AWS services such as Amazon EBS Snapshots use this approach to back up volumes for disaster recovery.

Log-based replication

Log-based replication tracks changes at the transaction level by reading database logs. Only specific changes, rather than the entire dataset, are replicated in near real time. This method is highly efficient and reduces network usage relative to snapshot-based replication. This approach is widely used in enterprise database management to replicate ongoing transactions.

Here are some common use cases for replication:

Database replication

You copy tables, schemas, and transactions across multiple instances. This approach is often used to maintain high availability or to perform read scaling (e.g., Amazon RDS Multi-AZ and Read Replicas).

Storage replication

This approach duplicates entire volumes or file systems to mitigate the effects of regional outages that might take an important service offline. Amazon EBS performs this type of replication.

Application data replication

You replicate user and session data across multiple environments. This is most commonly performed to improve application availability through better fault tolerance.

Cross-region replication

Copies data across AWS Regions to meet disaster recovery, latency, and compliance requirements.

Data replication ensures important data stays accessible even during network or system failures. It can also enhance system performance and user experiences by distributing workloads across multiple servers to help prevent bottlenecks. It also can help you bring data physically closer to global users to reduce latency.

High availability

Replicating data across multiple, geographically distributed systems helps you ensure it remains available to a global user base, regardless of what local disruptions might impact any one node.

Disaster recovery

By distributing copies of your data geographically, you can more easily restore service in the event of a regional failure in any one location. For example, if a fire or flood affects one data center, your failover mechanisms can redirect user requests to the next most available replica.

Performance optimization

Instead of routing all requests to a single database, replication allows you to distribute read traffic to multiple servers. Each request can go to the geographically closest server with the lowest workload, for instance.

Data locality

Keeping data closer to end users reduces latency in their time to access, meaning your applications can be more responsive. For example, by placing copies in edge locations or AWS Regions near key markets, you can improve performance for real-time applications such as gaming, financial trading, or customer-facing SaaS platforms.

Compliance and governance

Data privacy regulations may have requirements to store data in specific geographic jurisdictions, with specific security configurations, or for defined periods. Regulations that your organization may require compliance with include the European Union’s GDPR data privacy regulations and the United States' HIPAA healthcare data regulations. Keeping extra copies of data or applications within the relevant jurisdiction makes it easier to comply with such regulations.

Deploying a data replication strategy that balances cost, performance, and security requires careful planning. Organizations benefit from considering several factors to help identify the approach that best fits their business case.

Setting your Recovery Point Objective (RPO) and Recovery Time Objective (RTO) will help define your overall risk tolerance, which in turn helps determine your acceptable cost factors.

Recovery Point Objective (RPO)

RPO defines the maximum amount of data loss your organization can tolerate, and it is measured in time. Having a small RPO, meaning losses within seconds or minutes, might require you to perform synchronous or log-based replication. A larger RPO, on the timescale of hours, can allow you to use snapshot or asynchronous replication.

Recovery Time Objective (RTO)

RTO is defined as your organization’s maximum acceptable downtime after a disruption. If you have a low RTO, you will need access to more readily available and up-to-date replicas and have well-defined failover mechanisms.

Cost implications

Replication incurs costs for storage, data transfer, and compute. If you want to perform cross-region replication, it can also incur egress fees for moving data out of the local region.

Bandwidth utilization

Replication can take up a large amount of network resources, especially when moving data across regions or for large datasets. Strategies for optimizing bandwidth include compressing data, performing snapshot or incremental updates, and throttling transfer speeds. You can consider each of these strategies according to risk and cost.

Data consistency requirements

Risk tolerance and data consistency requirements will also impact the replication approaches you want to consider. For example, a mission critical database might need synchronous replication to ensure high transactional integrity, and might benefit from using a service such as Amazon Relational Database Service (Amazon RDS). Web analytics platforms might need high performance, but not necessarily the same level of real-time consistency, so some form of asynchronous replication can be more cost-effective.

AWS offers multiple solutions to help with your data replication needs across databases, storage, and applications:

• Amazon Elastic Block Store (Amazon EBS) offers the Amazon EBS Snapshots data replication tool – a point-in-time copy that can be used to enable disaster recovery, migrate data across regions and accounts, and improve backup compliance.
• Amazon Relational Database Service (Amazon RDS) offers highly available, durable relational databases deployed across up to three Availability Zones (AZs), for automatic failover.
• Amazon Simple Storage Service (Amazon S3) allows you to meet your recovery time objective (RTO), recovery point objective (RPO), and compliance requirements with S3's robust replication functionality, data protection with AWS Backup, and various AWS Partner Network solutions.
• AWS Database Migration Service (AWS DMS) enhances resilience through Multi-AZ redundancy and checkpoint-based recovery capabilities during migrations.
• AWS DataSync allows you to securely replicate your data into AWS object and file storage services, including Amazon S3, Amazon EFS, and Amazon FSx.

Get started with data replication on AWS by creating a free account today.