What change data capture does
An application database is often the authoritative home for customer accounts, orders, or product records. Other systems may need the same information in a different form: a search index for quick lookup, an analytics store for reporting, or a cache for fast reads. Change data capture, usually shortened to CDC, detects changes in the database and makes them available to those systems without repeatedly copying every row.
A CDC event typically identifies a table or collection, the affected record, and the kind of change: insert, update, or delete. It may also carry values from before or after the change. A consumer reads the event and updates its own copy. For example, when a product price changes, a search-index consumer can replace the price shown in search results. CDC moves data; it does not automatically make the destination consistent with the source at every instant.
Where the changes come from
One approach reads the database’s transaction log, the record the database uses to track committed changes. A CDC connector can follow that log and publish changes after they commit. This usually avoids asking the application to send a second message for every write, but it requires database-specific setup, permissions, and attention to how long the log remains available.
Another approach periodically checks rows for a modification timestamp or an increasing identifier. Polling can be simpler to introduce, but it can miss deletes and may miss intermediate updates between checks. Application-written events are a third option: the application records an event as part of its write workflow. That provides control over what an event means, but coordinating the database write with event publication needs care so one does not succeed while the other is lost.
A new destination also needs existing records, not just future changes. A common plan starts with a snapshot of current data, then applies changes that occurred during and after the snapshot. The handoff matters: if the pipeline has a gap between the snapshot and the change stream, records can be missed; if it overlaps without a way to handle duplicates, records can be applied twice.
A simple pipeline example
Imagine a small shop that stores products in a relational database and wants a separate search index. A CDC connector observes committed changes to the products table and sends events to a stream. A search consumer reads each event, uses the product ID as its index key, and writes the latest searchable fields. On a delete event, it removes that product from the index. The shop can rebuild the index from a fresh snapshot and then resume processing changes if the destination becomes unusable.
The stream between the connector and consumer is useful because the two systems do not have to run at exactly the same speed. It does, however, add an operating concern: if the consumer falls behind, search results may be stale until it catches up. A team should decide how much delay is acceptable and monitor both the connector and the consumer, rather than treating a running process as proof that data is up to date.
The reliability details that matter
Many CDC pipelines may deliver an event more than once, particularly after a restart or retry. Consumers should therefore be safe to rerun. Replacing an indexed document by its stable ID is often easier to make repeatable than incrementing a counter on every received event. If updates for the same record can arrive out of order, a version or source position can help the consumer reject an older change. Ordering across different records may not be necessary, but ordering for one record often is.
Deletes need an explicit plan. A destination that only processes inserts and updates will retain records that no longer exist in the source. Schema changes need a plan too: adding, renaming, or changing a column can affect event formats and consumer code. Test those changes with realistic events before deploying them. Also decide what happens when an event cannot be processed: repeated retries can stall progress, while skipping it can leave the destination incorrect. Keep enough information to investigate and replay failures.
Choosing and operating CDC responsibly
CDC is a good fit when another system needs an ongoing copy of database changes and a short propagation delay is acceptable. It is less attractive when a simple scheduled export meets the need, or when the receiving system cannot handle retries, gaps, and rebuilds. Start with one well-defined table and destination. Document the source of truth, the fields being copied, the acceptable delay, and the steps for checking whether a destination matches the source.
Finally, treat the change stream as sensitive data. Events may contain personal details or fields that a destination does not need. Limit captured fields where possible, restrict who can read the stream, and consider how long events and failed-message records are retained. A dependable CDC pipeline is not just a connector: it includes a safe initial load, consumers designed for replay, monitoring for lag and errors, and a tested way to recover when systems drift apart.
