Change data capture: keeping systems in sync
A change happens once. Every view that should reflect it does, in order, without duplication. This is the field that makes that true.
Change data capture, or CDC, is the practice of noticing every change in a source system the moment it happens and carrying that change to wherever else it needs to be reflected. We walk it the way the field is built: what it is, why it is hard, how a modern pipeline does it, what it unlocks, and where it is going. The harder cousin, two-way sync, gets its own treatment. The papers behind each idea are linked, so you can read the source yourself.
- Capture the committed change
Read the update from the database change log.
- Publish a change event
account: Acme · stage: Won - Update downstream views
The search index and reporting table consume the event.
Capturing every change, then carrying it in order.
A change is any insert, update, or delete in a source system. CDC captures each one as it happens and delivers it, in order, to other systems that need to reflect it.
Picture a salesperson's working memory. Email holds the thread. The calendar holds the next meeting. One CRM holds the deal, and a second CRM, a different team in a different region, holds another copy of the same account. A task list holds the follow-ups. Every one of these systems changes all day, and none of them tells the others. So the same contact shows up three times under slightly different names, a meeting that moved still reads as next Tuesday in one place, and a closed task is still open in another.
The atomic unit CDC works with is a change event: what row or record changed, what the new value is, often the old value too, and a position marker that fixes its order. The reliable way to produce that stream is to read the log the database already keeps. Databases write a durable, ordered record of every committed change, the write-ahead log, binary log, or redo log, so they can recover after a crash. Log-based CDC tails that same log. The changes are already ordered, already complete, and reading them adds little load to the source.
It helps to contrast the approaches a reader already knows. Polling and full reload re-reads everything on a timer and diffs it: simple, but slow, heavy on the source, and blind to anything that changed and changed back between polls. Dual writes have the application write to both systems itself: simple until one write succeeds and the other fails, and now the two systems disagree with no record of why.
Deal 482: Proposal → Won
Commit position: 1184- Search index
- Refresh the searchable deal.
- Reporting view
- Update won revenue.
- Task service
- Prepare the handoff workflow.
Carrying a change reliably is harder than carrying it once.
A change must not be lost, must not be applied twice in a way that corrupts state, and must still make sense when the same fact is edited in two places at once.
The log only holds recent history.
Transaction logs are trimmed, so a consumer that starts today cannot replay last year from the log alone. Bootstrapping a new view needs a full snapshot of current state, stitched to the live stream without missing or double-counting changes that land mid-snapshot.
Exactly-once is hard, and order matters.
A change must not be lost and should not be applied twice in a way that corrupts state. Most systems guarantee at-least-once delivery, then lean on idempotent apply plus per-key ordering, so a replayed event is harmless.
The source keeps changing shape.
The source adds a column, renames a field, changes a type. The pipeline has to carry those schema changes through without breaking the downstream consumers that depend on the old shape.
Two valid edits can collide.
When the same fact can be edited in two places, a rep updates the deal stage in CRM A while an automation updates it in CRM B, two changes arrive at once. The system has to decide an order, detect that they touch the same thing, and reconcile it. Naive last-writer-wins is simple but silently discards one edit.
Is "Bob Smith" the same as "Robert Smith"?
"Robert Smith" in CRM A and "Bob Smith" in CRM B may be one person. Syncing changes is only useful once the systems agree what is the same record. That is entity resolution, which has its own treatment. CDC delivers the changes, resolution decides which entity they belong to. See Knowledge graphs.
Watching cannot strain the system being watched.
Re-querying everything on a timer puts real load on a live database. Capture has to stay light, which is the practical reason log-based methods, reading what the database already wrote, won out over constant polling.
A pipeline moves a change in named stages.
Capture the change, backfill the history, carry it on an ordered log, apply it safely, and reconcile when edits flow both ways.
- 01Capture
Read committed changes.
- 02Backfill
Coordinate a baseline snapshot.
- 03Transport
Retain an ordered stream.
- 04Apply
Handle retries safely.
- 05Reconcile
Resolve cross-system conflicts.
Five jobs in a row, each doing one thing, turning a change in one system into a synced view in another.
1Capture
Read the change the database already recorded.
Log-based capture tails the transaction log, a common method that stays light on the source. Named honestly for contrast: query-based capture polls a "last updated" column, which is simple but misses deletes and intermediate states, and trigger-based capture writes change rows from database triggers, which is accurate but adds load to the source.
The seminal end-to-end account of pulling committed changes and delivering them to many consumers is Databus (Das et al., 2012). Open-source systems such as Debezium, built on the Kafka Connect framework, are widely used implementations of log-based capture. They are named here as examples of the field, not as endorsements.
| Position | Operation | Record |
|---|---|---|
| 1184 | Update | deal:482 · stage = Won |
| 1185 | Insert | task:991 · send quote |
| 1186 | Delete | contact:483 |
2Snapshot and backfill
Give a new view the history the log no longer holds.
Because the log only keeps recent changes, a new consumer needs a full snapshot of current state, then a clean hand-off to the live log with no gaps and no duplicates. The hard part is the seam: changes keep landing while the snapshot runs.
DBLog (Andreakis & Papapanagiotou, 2020) solves this with a watermark-and-chunk technique: take the snapshot in pieces, lock-free, interleaved with the live stream, using watermarks to mark exactly where the two join. History and live changes end up stitched together with nothing missed and nothing counted twice.
- Begin the snapshot
Record the position needed to coordinate the backfill.
- Copy existing rows
Continue retaining live changes while reading historical state.
- Reconcile the boundary
Apply the connector’s watermark or consistency protocol.
- Continue from the log
Process subsequent changes without losing the overlap.
3Transport
The log is the backbone.
Change events travel as a durable, ordered, replayable log. Partition the log by key so a single consumer keeps per-key order while many consumers scale out. The plain-spoken case that an ordered, append-only log is the unifying abstraction for keeping systems in sync is Kreps, "The Log" (2013).
The design of a partitioned, durable, replayable log that became a common transport for change streams is Kafka (Kreps, Narkhede, Rao, 2011). Replayability is what lets a new or recovered consumer rebuild its view from a known position, rather than starting from nothing.
| Consumer | Last applied | Next event |
|---|---|---|
| Search | 1186 | 1187 |
| CRM replica | 1184 | 1185 |
| Reporting | 1182 | 1183 |
Readers progress independently and can replay retained events.
4Apply, and the dual-write fix
Apply each event once, safely, even on replay.
A consumer applies each event to its own view idempotently, so applying the same event twice leaves the view unchanged and a replay is safe. That is what makes at-least-once delivery workable in practice.
The outbox pattern, also called transactional-log-tail, solves the dual-write trap. The application writes its change and an outbox event in one local transaction, and CDC ships the outbox. The two systems can no longer disagree, because there was only ever one write. Two writes can drift apart. One write, then captured, cannot.
Two independent writes
- Database write
The deal update succeeds.
- Message send
A failure leaves the destination uninformed.
Transactional outbox
- One transaction
Write the deal update and an outbox event.
- Capture and deliver
Publish the committed outbox event; deduplicate retries downstream.
5Reconcile, the two-way case
When two edits collide, a stated rule decides.
When changes flow both ways, the field has to decide what "current" even means when two truths arrive at once. Options run from simple to rich. Last-writer-wins by timestamp or logical clock is simple but lossy: it silently discards one edit. Application-defined merge procedures detect and resolve conflicts per write, the model introduced in Bayou (Terry et al., 1995).
The principled end of the spectrum is conflict-free replicated data types (Shapiro et al., 2011), data structures designed so concurrent edits always merge to the same result with no central coordinator. They fit some data shapes cleanly, counters, sets, text, and others awkwardly, so they are a strong tool rather than a universal answer. Across all of these, the goal is the same: detect the conflict and resolve it by a stated policy, instead of letting one side disappear quietly.
| Source | Proposed stage |
|---|---|
| CRM A | Won |
| CRM B | Negotiation |
For example, require review when one change closes a deal and another reopens negotiation.
One current, de-duplicated view, kept field by field.
At the field level: a connected knowledge layer where email, calendar, more than one CRM, and a task list reflect the same change soon after it happens, instead of drifting apart between batch jobs.
Picture the salesperson's view again. The same contact, shown three different ways across systems, settles into a single current record. The meeting that moved reads as moved everywhere. The closed task reads as closed everywhere. That is what a working sync layer is for, and the hard parts above, snapshots, ordering, conflicts, are exactly what decide whether the view stays honest.
The duplicates collapse.
Once systems agree what is the same record, "Robert Smith" and "Bob Smith" stop being two contacts. The de-duplication rests on entity resolution. See Knowledge graphs.
Lower load on the source.
Reading the log the database already writes is far lighter than re-querying everything on a timer, so the system being watched is not strained by the watching.
Replay and recovery.
Because the change history is an ordered log, a new view can be built from scratch and a broken one rebuilt by replaying from a saved position.
Honest two-way edits.
With reconciliation rules in place, a fact can be edited in more than one place without one side silently winning, because conflicts are detected and resolved by a stated policy.
This page describes the field, not any one product's plumbing. How a given tool unifies email, calendar, CRMs, and tasks is its own engineering, and is out of scope here.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Capture once, fan out to many.
The log-as-backbone pattern is increasingly a common way systems stay in sync, rather than point-to-point wiring between every pair of systems.
Backfills that do not freeze the source.
Lock-free, chunked, pausable backfills, the DBLog line of work, are becoming the norm, so a new consumer can join without locking up the system it reads from.
Toward effectively-exactly-once.
Continued movement from at-least-once toward effectively-exactly-once apply, through idempotency keys and transactional outboxes.
Lose less when edits collide.
Wider use of CRDTs and richer merge logic, so two-way sync resolves more conflicts automatically and surfaces the rest, instead of defaulting to last-writer-wins.
Carry the shape, not just the value.
Carrying schema changes through the stream automatically, so downstream views do not break when the source adds, renames, or retypes a field.
The papers behind this page.
- Kreps (2013). The Log: What every software engineer should know about real-time data's unifying abstraction. LinkedIn Engineering.
- Kreps, Narkhede & Rao (2011). Kafka: a Distributed Messaging System for Log Processing. NetDB workshop.
- Das et al. (2012). All Aboard the Databus! LinkedIn's Scalable Consistent Change Data Capture Platform. ACM SoCC. doi:10.1145/2391229.2391247.
- Andreakis & Papapanagiotou (2020). DBLog: A Watermark Based Change-Data-Capture Framework. arXiv:2010.12597.
- Terry et al. (1995). Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System. ACM SOSP. doi:10.1145/224056.224070.
- Shapiro, Preguiça, Baquero & Zawirski (2011). Conflict-free Replicated Data Types. INRIA Research Report / SSS.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
You talk, and your systems stay current behind you. See it on a real call.
No app required. Just your phone number.