What is Change Data Capture?

Change data capture (CDC) is a data synchronization pattern that sends log or database changes, such as insert, update, and delete, to other systems in near-real time. CDC is useful in streaming data pipelines, logging, and transactional database operations. CDC is designed to help save bandwidth and time while maintaining consistency across systems.

Many modern data stacks rely on consistent delivery from their source systems. If entire datasets were transferred every time these systems were to update, it would consume huge portions of network bandwidth. Full-table scans would also impact database performance, worsening the overall efficiency of these data systems.

Change data capture solves these issues by incrementally capturing updates within source systems. It helps enable real-time data availability, as you don’t have to rely on scheduled batch jobs while also maintaining complete consistency and reducing load on source systems.

CDC is commonly used to feed events to streaming systems and within loosely coupled microservices, as it simplifies data replication across these environments. Especially as real-time data delivery has become more popular in data infrastructure, CDC is now a common method to provide reliable and scalable data flow.

There are several benefits of using change data capture, which make it a useful strategy for businesses to employ.

Real-time data availability

By synchronizing data from source systems almost immediately, CDC helps provide downstream systems such as analytics platforms with up-to-date information. It does this without the need for polling, which makes it a much more scalable and low-stress data sync system.

Reduced load on source databases

Change data capture only transmits changed rows, rather than conducting full-scale replications. It also performs iterative, continuous transmission, rather than periodic batch loads. Both of these work together to reduce the overall load on source databases.

Event-driven architecture support

CDC turns database changes into a stream of events that multiple services can consume. Especially for event-driven architectures, this helps to decouple systems, making it much easier to scale applications or introduce new consumers without having to restructure the source system.

Reliable audit trail

Change data capture provides an ordered sequence of changes, making it easy to record updates, deletes, or inserts. For any auditing or debugging, CDC supports the compliance process and makes it easy to organize evidence.

Simplified data synchronization

By streaming to multiple consumers, CDC makes it easy for data warehouses, data lakes, replicas, or any other connected systems to stay in sync. You won’t need to build and run any fragile ETL scripts, instead relying on CDC techniques for simplified data synchronization.

Change data capture works by monitoring a source system for changes, converting those changes into events, and then publishing them into a data pipeline. Any downstream system, like a data warehouse or analytics platform, can then consume these events and apply the changes they hold to maintain consistency.

The actual internal processes that happen during this process depend on the type of CDC that you implement. Here are the most common approaches to implementing CDC.

Log-based CDC

Log-based CDC directly reads from database transaction logs. Depending on the system, this could be WAL (write-ahead log) in PostgreSQL, the binary log (binlog) in MySQL, or redo logs in Oracle. A log-based approach to CDC has minimal impact on the source database and much less overhead than other systems. As it works on the database engine level, it doesn’t have the same overhead as application-layer mechanisms while still providing changes in the exact order as they occur.

Trigger-based CDC

Trigger-based CDC uses a range of database triggers, which will execute whenever data manipulation operations happen. Any insert, update, or delete will trigger a write of the corresponding record to a separate table. These alternative tables are called audit tables or shadow tables. The system uses these tables to track changes within the source system.

With the additional steps and infrastructural elements, this form of CDC requires more overhead. It also creates tighter coupling between application logic and data capture mechanisms.

Timestamp-based CDC

Timestamp-based CDC uses a specific timestamp to qualify any records that have been modified after its specific time. It’s a simple form of CDC, but it requires a reliable timestamp column and may miss delete operations. Additionally, it may begin to produce inconsistencies if clocks aren’t synchronized or if an update happens outside of the expected pattern. While simple, it’s prone to drift, making it a less commonly used form.

Diff-based (snapshot) CDC

Diff-based CDC compares full table or database snapshots to clearly identify if any changes have occurred. Diff-based approaches are resource-intensive, as they require a full scan of the source environment, creating a high resource cost. This system is rarely used in production streaming contexts, but is common in batch extract, transform, load (ETL) pipelines.

A CDC pipeline has several components that work together to accurately transmit changes from source systems to target consumers.

Source connector

Source connectors are the components that interface with sources and capture any changes to them. Typically, this uses strategies like reading transaction logs or monitoring triggers, but it depends on the type of CDC pipeline you’re using.

Change event stream

Change event streams are the transport layer that takes detected changes in source databases and transmits them. Most of the time, distributed messaging systems like Apache Kafka or managed streaming services cover this component.

Schema registry

Schema registries track versions and enforce schema rules, helping to manage any schema evolution in source systems. You need a schema registry to make sure that data remains compatible between producer systems and consumers.

Sink connector/consumer system

A sink connector delivers the change events to consumer systems by writing them into the system. It’s the final step in CDC, updating records in a consumer system to match the source database.

Here are a few common use cases of CDC for real-time data movement and consistency.

Database replication and synchronization

You can replicate data and synchronize across multiple databases consistently with CDC. By propagating changes as they occur to multiple destinations, you can keep many databases in sync.

Real-time analytics and data warehousing

Real-time analytics systems and dashboards rely on up-to-date and timely data. CDC continuously integrates operational data into these systems, ensuring they’re up-to-date without the need to wait for batch processes to run.

Event-driven microservices

CDC helps in event-driven architecture by converting changes in databases into events, which these services can then react to in real-time without directly coupling to the source system. This enables agility and timeliness, without adding architectural burden.

Cache invalidation

Cache invalidation, where stale data is removed from a cache when the source data changes, is made more efficient with CDC. CDC creates events whenever data is inserted, updated, or deleted, allowing downstream systems to invalidate the cache or update it with new values. The systems receive low-latency access without having to periodically rely on cache refreshes.

Audit logging and compliance

When auditing your data processes, CDC is useful, as it creates a record of all the changes that you make to data over time. Especially in highly regulated environments, this comprehensive record that includes what changes happened, when they occurred, and who made each modification will make auditing much more straightforward.

CDC naturally supports your auditing and compliance processes by capturing every insert, delete, and update as a unique event. You can review the history of these events to get a clear picture of all changes.

Zero-downtime database migrations

When migrating data from one system to another, you can use CDC to continuously replicate changes from the source to the target system throughout the migration process. By initially copying a snapshot of the data and relating it to the target, you can then keep the target in sync by streaming any changes in the source system.

This lets you fully synchronize the systems before switching over to the new database, all without incurring downtime or creating disruptions.

CDC and ETL both move data from one place to another. However, they take different approaches, have different benefits, and are mainly employed in distinct use cases.

CDC captures changes and propagates them to downstream systems as the changes occur, which provides a low-latency method for data movement. ETL systems instead process at batch intervals, which introduces latency but also makes it more effective for large-scale bulk transformations and processing.

CDC also distributes load evenly by streaming incremental updates, while ETL will create periodic spikes of demand due to processing many things at once. Due to the continual need to move information with CDC, it is normally more complex to integrate, while ETL doesn’t require streaming infrastructure and is fairly straightforward.

AWS offers a range of services to help you implement CDC pipelines at scale. Choose the managed solution for your use case:

• Amazon Athena is a serverless interactive query service that simplifies data analysis and is CDC compatible for near-real-time querying.
• Amazon Kinesis Data Streams ingests and collects terabytes of data per day from application and service logs, clickstream data, sensor data, and in-app user events to power live dashboards, generate metrics, and deliver data into data lakes.
• Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a streaming data service that manages Apache Kafka infrastructure and operations, making it easier for developers and DevOps managers to run Apache Kafka applications and Apache Kafka Connect connectors on AWS.

Get started with implementing change data capture on AWS by creating a free account today.