Incremental sync: webhooks, polling, and sync tokens.
Many systems change all day. Keeping one current, de-duplicated view of them is its own field. Here is how it works, in plain language.
Incremental sync is the practice of keeping a local view agreeing with one or more remote systems by transferring only what changed, rather than re-reading everything each time. We walk it the way the field is built: what it is, why it is hard, how a modern system does it, what it unlocks, and where it is heading. The papers and specifications behind each idea are linked, so you can read the source yourself.
The three ideas to carry: the stored bookmark, push versus pull, and conflict handling.
- Read changes after 104
One meeting moved, one contact updated, one task completed.
- Apply and save checkpoint 107
The next sync resumes after this point.
Transfer only what changed, not everything again.
The plain framing: instead of "send me your whole mailbox again," you say "send me only what changed since this bookmark."
The bookmark is the heart of it. Google Calendar's sync token, Gmail's history id, Microsoft Graph's delta token, and Salesforce's replay id are four names for the same idea: a cursor that marks where you last got to. You store it, and next time you get back just the new, changed, and deleted items, plus a fresh bookmark.
Two separate questions often get confused. When do I find out a change happened, which is timing: polling versus webhooks. And how little do I have to transfer to catch up, which is efficiency: full versus incremental. A system can mix and match. A webhook can wake a consumer, after which it does an incremental pull to get the actual changes.
The running example across this page: a salesperson's email, calendar, two CRMs, and a task list, merged into one current, de-duplicated view where a change in any source propagates and the same contact is one record, not several. The example illustrates the problem. It is not a description of any product.
Existing copy
Records through checkpoint 104Delta
105: add
106: update
107: deleteApply the changes and store checkpoint 107.
No timer setting is both cheap and current.
Keeping many systems honest with each other runs into cost, timing, identity, and concurrency at once.
Re-downloading everything does not scale.
Re-reading an entire mailbox or account list on a timer to find the few things that moved is mostly wasted work. The cost grows with the size of the data, not the size of the change, and it still lags by the length of the timer.
Polling forces a bad tradeoff.
Poll often and you burn rate limits and money on mostly-empty checks. Poll rarely and the view goes stale. There is no single timer setting that is both inexpensive and current.
Webhooks move the hard part to you.
A push notification is timely, but now you run an always-on receiver. Providers deliver at-least-once, so duplicate and out-of-order notifications arrive and must be tolerated. Notifications can also be lost, so a periodic reconciliation sweep stays necessary.
Sync tokens do not last forever.
Gmail returns HTTP 404 when a start history id is too old. Google Calendar returns HTTP 410 Gone when a sync token is invalidated. The system must always be able to fall back to a full sync and rebuild the baseline.
Deletes are easy to miss.
A change feed has to report not just new and edited items but also deleted ones, or the local view keeps tombstones forever. The good change APIs explicitly return deletions so the consumer can remove them.
The same person appears many times.
Across two CRMs, an email thread, and a calendar invite, one human shows up as several records with different keys. Before you can sync a change to "the contact," you have to decide which records are the same entity. That step is upstream of sync.
Two-way sync invites conflicts.
When changes can originate on any side, two systems can edit the same field at nearly the same moment. Without a rule for who wins, you get lost updates or an echo loop. Clocks on different machines disagree, so "most recent wins" is not as simple as it sounds.
Sync once in full, then carry only the difference.
Establish a baseline, store a bookmark, pull only changes forever after, and reconcile what arrives on more than one side.
- 01Baseline
Read the initial dataset.
- 02Checkpoint
Store the source position.
- 03Delta
Fetch subsequent changes.
- 04Repeat
Apply safely and advance.
- 05Recover
Rebuild if the cursor expires.
The baseline-then-incremental loop is the spine of Gmail, Google Calendar, and Microsoft Graph delta query alike.
1The baseline-then-incremental loop
One full sync, then only the changes.
Do one full sync to populate the local store and record the returned token. From then on, send the token back and receive only the new, changed, and deleted items since then, plus a fresh token to store. Repeat. If the token is rejected as expired, do a full sync again and carry on.
This loop is documented the same way across Gmail, Google Calendar, and Microsoft Graph delta query. Each leads with the same shape: a baseline once, cheap partial syncs forever after, and a full re-sync only as the safety net.
- Initial sync
Fetch the baseline dataset.
- Save checkpoint
Record the source’s continuation or sync cursor.
- Fetch changes
Request the next page after that position.
- Apply and checkpoint
Commit changes and progress safely before continuing.
2What the bookmark actually is
An opaque cursor, not a timestamp.
Tokens are opaque cursors, not timestamps you should parse. Microsoft Graph bakes the original query parameters into the token, so each round reuses the same parameters. Google Calendar similarly restricts which query parameters may change between incremental calls. The token is a promise from the server about where you were, not data you own.
Some systems express the same bookmark as a stream position. Salesforce Change Data Capture delivers record-change events on an event bus, each tagged with a replay id marking its place in the stream. A subscriber stores the last replay id and resubscribes from there to catch up on anything missed while disconnected, within a retention window of 72 hours.
| Source | Progress marker |
|---|---|
| Google Calendar | nextSyncToken |
| Gmail | historyId |
| Microsoft Graph | delta token / delta link |
| Salesforce events | Replay ID |
3Webhooks: be told, do not ask
Ask on a timer, or be told on a change.
Instead of polling, the consumer registers a callback URL and the provider sends an HTTP POST when something changes. WebSub is the standardized, open version of this pattern, with a publisher, a subscriber, and an intermediary hub that verifies subscriptions and signs deliveries.
A common production pattern is hybrid. The webhook is a low-latency nudge that says "something changed," and the consumer responds by doing an incremental pull with its stored token to fetch the actual, authoritative changes. The push gives timeliness. The pull gives correctness. Public guidance, such as the Hookdeck webhooks guide, frames push and pull as complementary tools with tradeoffs, not a winner and a loser.
Polling
- Timer fires
Ask whether anything changed.
- Fetch delta
Use the stored cursor.
Webhook-assisted
- Notification arrives
Treat it as a prompt to check the source.
- Fetch delta
Use the stored cursor for authoritative changes.
4Idempotency and reconciliation
Expect duplicates, keep a safety sweep.
Because deliveries are at-least-once and can be lost, robust syncs do two things. They deduplicate by a stable change id, so reprocessing the same notification is harmless. And they run a slower full or token-based reconciliation on a schedule, to catch anything the push path dropped.
Webhooks reduce how often you poll. They rarely remove the safety sweep entirely. A lost notification is invisible until the periodic reconciliation finds the gap, which is why the backstop stays in the design even when push is working well.
Use stable source identifiers and safe writes.
Periodically reconcile the local copy with the source.
5Resolving conflicts on two-way sync
A deterministic rule, not luck.
When both sides can write, the system needs a deterministic rule for concurrent edits. The classic systems answer comes from Bayou (Terry et al., 1995), which paired application-defined dependency checks, to detect that the precondition for an update no longer holds, with merge procedures, a per-write rule that resolves the conflict deterministically, so that every replica converges to the same result.
A complementary modern approach is conflict-free replicated data types (Shapiro et al., 2011), which design the data so concurrent updates commute and merge automatically without a central referee. Field-level "last writer wins" is the simplest rule and the easiest to get wrong, because it silently drops one edit and depends on clocks agreeing.
| Source | Phone update |
|---|---|
| CRM A | 555-0102 |
| CRM B | 555-0199 |
Use version metadata and the chosen conflict policy.
6Identity resolution feeds sync
You sync changes to entities, not to rows.
Before a change to "this contact" can propagate, the layer must map each source record to a single entity, matching on email, domain, name, and so on. De-duplication is upstream of, and distinct from, the change-transfer mechanics, but the two have to cooperate.
The honest limit is that matching can be wrong, and a bad match propagates a change to the wrong entity. Identity resolution is its own problem, treated in its own right. Here it is enough to note that the change feed delivers raw rows, and something has to decide which rows are the same human before a sync is meaningful.
| Source key | Evidence | Destination |
|---|---|---|
| CRM A:17 | Verified ID 482 | Contact 482 |
| CRM B:91 | Verified ID 482 | Contact 482 |
| Import:6 | Similar name only | Unresolved candidate |
A current view without brute force.
Incremental sync keeps a unified view close to live while transferring a fraction of the data, because the cost tracks the size of the change, not the size of the dataset.
The relatable payoff is in the running example, as problem illustration only and not a product claim: a contact's new phone number entered in one CRM appearing in the unified view, a meeting reschedule reflected without a manual re-import, a task marked done in one place no longer showing as open elsewhere, two account records recognized as one company. These are the kinds of unify-and-sync problems the field's mechanics are built to address.
Close to live, at a fraction of the data.
The cost of staying current tracks the size of the change, not the size of the dataset, so a unified view can stay near real time without re-scanning everything.
Lighter load on the systems of record.
Token-based pulls and webhook nudges replace constant full scans, which respects rate limits and keeps the source systems healthy.
Resilience to disconnection.
Stored cursors and retained change streams let a consumer go offline and catch up on return, rather than starting over from a full sync each time.
De-duplicated, not near-copies.
When identity resolution and sync cooperate, one human is one record across sources, and a change made anywhere lands on the right entity.
Everything here describes the field. It is not a description of how any one product syncs data, and it says nothing about Prism internals or plumbing.
The direction the field is heading.
We describe the direction, and the honest limits, not a roadmap of our own.
Durable, replayable event streams.
Vendor change feeds keep moving from periodic pull toward durable, replayable streams, such as Salesforce's Pub/Sub API delivering change events with replayable positions, so consumers resume from a marker instead of re-scanning.
CRDTs spread beyond editors.
Conflict-free replicated data types continue to spread from collaborative editors into general data sync, because they let independent replicas merge concurrent edits deterministically without round-trips to a central authority.
The receiver side, made reusable.
Open protocols such as WebSub, which reached W3C Recommendation status, and managed webhook delivery services aim to make verification, retries, signatures, and deduplication reusable rather than hand-rolled per integration.
Tokens still expire.
Bookmarks do not last forever, so periodic full re-syncs stay part of the design. A 404 or a 410 Gone is a signal to rebuild the baseline, not a failure to engineer away.
Lower cost is not correctness.
Deliveries duplicate, reorder, and occasionally vanish. Clocks disagree, so naive "latest wins" can lose real edits. De-duplication can mismatch. Incremental sync lowers cost and lag, which is why conflict rules and reconciliation still matter.
The papers and specifications behind this page.
- Terry, Theimer, Petersen, Demers, Spreitzer, Hauser (1995). Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System. SOSP 1995.
- Shapiro, Preguiça, Baquero, Zawirski (2011). Conflict-free Replicated Data Types. Inria RR-7687, SSS 2011.
- W3C. WebSub. W3C Recommendation.
- Google Workspace. Synchronize clients with Gmail. Gmail API guide.
- Google Workspace. Synchronize resources efficiently. Google Calendar API guide.
- Microsoft Learn. Use delta query to track changes in Microsoft Graph data.
- Salesforce Developers. Subscribe with Pub/Sub API · Change Data Capture Developer Guide.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Many systems, one current view, kept honest with each other. See what that feels like on a real call.
No app required. Just your phone number.