Exactly-once and idempotent writes.
The same update arrives three times. The calendar still shows one meeting. Here is the field that makes that happen, in plain language.
When data flows between systems, messages get retried, reordered, and duplicated. That is normal. Two ideas from data engineering keep a unified view honest anyway: idempotent writes, which you can apply many times for the same result, and exactly-once semantics, which makes the effect of a message land once even when the message does not. We walk it the way the field is built: what it is, why it is hard, how it works, what it makes possible, and where it is going. The papers behind each idea are linked, so you can read the source yourself.
The three ideas to carry: at-least-once delivery, idempotency keys, and convergent merge.
call-204OriginalAppliedcall-204RetryAlready appliedcall-204Retry againAlready applied
Make a write land once, even when the message arrives many times.
Two ideas do most of the work: idempotent writes, and exactly-once semantics. Deduplication is the machinery under both.
An idempotent write is an operation you can apply more than once and get the same result as applying it once. "Set the Acme call to Thursday" is naturally idempotent: run it five times, the call is still on Thursday. "Add a meeting" is not, because running it five times makes five meetings. A lot of the craft is turning the second kind into the first.
Exactly-once semantics is the guarantee that the effect of a message lands once, even if the message itself is delivered many times. It is a property of the outcome, not of the wire. The honest framing the field uses: you usually get at-least-once delivery, and you add idempotency or deduplication on top to reach exactly-once in effect. Deduplication is recognizing that two deliveries, or two records, refer to the same thing, and collapsing them to one.
To make this concrete, picture one running example, used only to illustrate the problem. A salesperson's working day is spread across five places at once: email, a calendar, two CRMs because the company switched systems and never finished the move, and a task list. The goal is one current, de-duplicated view, with a change made in any one tool reflected in the others. That is two-way sync, and the same meeting can arrive from the calendar and from a CRM, the same contact can live in both CRMs, and a phone edited offline syncs later. The question this page answers is how one true copy survives all that.
The receiver recognizes all three deliveries as the same operation.
Duplicates and reorderings are the normal weather of connected systems.
None of these are bugs. They are what happens whenever data flows between systems that can each drop, retry, or go offline.
Duplicates are unavoidable.
A sender that does not hear an acknowledgment cannot tell whether its message was lost, or only the acknowledgment was lost. The safe move is to send again. At-least-once delivery is the realistic default, so the receiver has to expect repeats.
Reordering and delay are normal.
Updates do not arrive in the order they happened. An offline phone replays yesterday's edits today. "Moved to Thursday" can land before "moved to Friday" or after it, and the system has to end in the same state either way.
"Create" is not repeat-safe.
A retried "add this meeting" or "log this contact" makes a second copy. Without a stable identity for the change, the receiver has no way to know it has seen this one before. This is where naive sync produces the three-Thursday-meetings problem.
One write often means several writes.
Recording a meeting might touch a contact, an activity log, and a task at once. If the process dies halfway, you get a contact updated but no task: a partial state that is worse than doing nothing. Landing a group of writes all-or-nothing is its own problem.
Two-way sync invites conflicts.
When the same record is edited in two systems before they reconcile, something must decide the merged value, and it must reach the same answer regardless of arrival order. Otherwise two copies that have seen the same edits can still disagree, which defeats the point.
Dedup needs a definition of identity.
Too strict and you keep duplicates ("Acme Corp" versus "Acme Corporation"). Too loose and you merge two genuinely different people. Identity is a design decision, not a given, which is why matching records is its own field.
Give every change an identity, then absorb the repeats.
Stable keys make repeats safe. Atomic commits make groups of writes safe. Snapshots make crashes safe. Convergent merge makes two-way edits settle.
1Stable identity for every change
An idempotency key, then upsert by key.
The sender attaches a unique, stable id to each operation. The receiver remembers ids it has already applied and absorbs any repeat: a retried request with the same key is a no-op, so "send again to be safe" stops being dangerous. This is the workhorse pattern behind idempotent writes, and the plain-language case for it is Helland (2012).
A close cousin is the upsert. Writing "this record, identified by this key, now has these values" is repeat-safe by construction: re-running it changes nothing. Turning "create" into "upsert by key" is the most common way to make a non-idempotent operation idempotent.
- Receive operation 7f3a
Check whether this key already has a recorded outcome.
- Apply once
Write the change and its key in the same atomic boundary.
- Retry arrives
Return the recorded result without repeating the effect.
2Spotting a repeat
Deduplicate on a sequence number or a fingerprint.
One way to recognize a repeat is a per-sender sequence number, so the receiver drops anything it has already seen. This is exactly how an idempotent producer works in a log system: each message carries a sequence number per sender, and the broker rejects a duplicate. The reference design is Confluent's account of how Kafka does it (2017).
When no clean id exists, another option is a fingerprint of the content. Either way, deduplication is the shared machinery: recognize "I have seen this before," and collapse it to one. How sharp that recognition is depends entirely on how identity is defined.
| Sender | Sequence | Action |
|---|---|---|
| A | 41 | Apply |
| A | 41 again | Duplicate: skip |
| A | 42 | Apply |
| B | 41 | Different sender: evaluate separately |
3Groups of writes, all-or-nothing
Commit related writes atomically.
To avoid half-applied state, related writes are committed all-or-nothing: either every write in the batch is visible or none is. Log systems do this with a transactional commit coordinated across destinations. Stream processors reach the same guarantee a different way, by checkpointing their state and input positions together, so on recovery they resume from a consistent point and each input is reflected once. MillWheel (Akidau et al., 2013) is a foundational example, writing produced records and state in one atomic step and deduplicating retries so user code can be non-idempotent yet behave idempotently.
- Update contact
Pending inside the transaction.
- Append activity
Pending inside the same transaction.
- Create follow-up task
Pending inside the same transaction.
The transaction defines which effects are atomic.
4Surviving a crash
Checkpoint, then replay from the snapshot.
Stream systems take consistent snapshots of in-flight state without stopping the flow, then on failure roll back to the last snapshot and reprocess. Because the snapshot and the input positions move together, records before the snapshot are reflected exactly once and records after it are reprocessed cleanly. This is the asynchronous barrier-snapshot idea, introduced in Carbone et al. (2015) and used in several modern engines.
Time-based data needs one more thing. For meetings, a system has to know when it has probably seen everything for a window, so it can finalize without double-counting late arrivals. Event-time processing with watermarks, framed in The Dataflow Model (Akidau et al., 2015), separates "what happened when" from "when we processed it."
- Checkpoint
- State through event 43, with input position 43.
- Failure
- Event 44 was in flight.
- Recovery
- Restore the checkpoint and replay from event 44.
A sink must also prevent duplicate visible effects during replay.
5Two-way edits that settle
Make merges converge.
When the same item is edited in two places, the system needs a rule that always settles to the same value no matter the arrival order. Conflict-free replicated data types are one principled answer: data structures whose merge operation is commutative and associative, so replicas that have seen the same set of updates always agree, with no central coordinator. The defining paper is Shapiro et al. (2011). Simpler schemes (last-write-wins by timestamp, or per-field merge) are common, but they have sharper edges, as the limits below note.
CRM A
title: VP OperationsCRM B
phone: 555-0148Keep both field updates under the defined merge rule.
Once writes are idempotent, retries stop being scary.
At the field level: safe retries, a de-duplicated view across many sources, two-way sync that settles, and pipelines that recover cleanly from failure.
These are the unify-and-sync problems that exactly-once and idempotent writes are built to frame. The running example, a salesperson's email, calendar, CRMs, and tasks pulled into one current view, is an illustration of the problem, not a description of any one product. What the field gives you is a set of guarantees you can reason about.
Retries become safe.
Once writes are idempotent, a sender can resend freely on any doubt and the receiver stays clean. That removes a whole class of fragile coordination.
One de-duplicated view.
A meeting that appears in the calendar and in a CRM resolves to one entry. A contact that lives in two CRMs resolves to one record. The clean view stops fighting the duplicates.
Two-way sync that settles.
Convergent merge lets edits happen in more than one place and still reach one agreed state, rather than two systems quietly drifting apart.
Recoverable pipelines.
Checkpointing and atomic commit mean a crash mid-sync resumes without duplicating or dropping work, so the unified view stays trustworthy through failures.
This page describes the field. It makes no claim about how any particular product, including ours, builds or syncs its data. Where the example mentions email, calendars, CRMs, and tasks, it is naming a familiar version of the problem, nothing more.
The direction the field is heading.
We describe the direction, and the honest limits, not a roadmap of our own.
Exactly-once by default.
What once required careful hand-rolled dedup is increasingly a built-in mode in log and stream systems, with transactional writes and checkpointing as standard features rather than expert add-ons.
Richer, lighter merges.
Work on conflict-free replicated data types keeps expanding the catalog (sets, counters, sequences, maps, text) and reducing its overhead, making coordination-free two-way sync more practical for everyday records.
Dedup as a first-class concern.
Better entity resolution, deciding when two records are the same real-world thing, tightens the "same" question dedup depends on, especially across systems that name things differently.
Effects that escape the boundary.
Exactly-once is a guarantee about effects inside a defined boundary, not the whole world. A side effect that escapes the system, an email actually sent or a webhook fired, can still happen twice unless it too is made idempotent.
Merges that drop real edits.
Last-write-wins can silently discard a genuine edit. Dedup that is too loose merges distinct records, too strict it keeps duplicates. These approaches lower the cost of duplicates and conflicts. They do not remove the need to define identity and to check results against reality.
The papers behind this page.
- Helland (2012). Idempotence Is Not a Medical Condition. ACM Queue, vol. 10 no. 4.
- Narkhede (2017). Exactly-Once Semantics Are Possible: Here's How Kafka Does It. Confluent (KIP-98).
- Akidau et al. (2013). MillWheel: Fault-Tolerant Stream Processing at Internet Scale. VLDB.
- Carbone et al. (2015). Lightweight Asynchronous Snapshots for Distributed Dataflows. arXiv:1506.08603.
- Akidau et al. (2015). The Dataflow Model. VLDB.
- Shapiro, Preguiça, Baquero, Zawirski (2011). Conflict-Free Replicated Data Types. SSS.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
One current view of every meeting, contact, and next step, kept honest as data flows in from many places. See it on a real call.
No app required. Just your phone number.