Event sourcing and the unified log.
Many drifting copies go in. One current, de-duplicated view comes out. Here is the field that makes that happen, in plain language.
Email in one place, calendar in another, one or two CRMs holding the deal, a task list somewhere else again. The same account in all of them, spelled differently, none of them agreeing. Event sourcing and the unified log are how the field turns that into a single connected view. We walk it the way the field is built: what it is, why it is hard, how a system does it, what it unlocks, and where it is heading. The papers and essays behind each idea are linked, so you can read the source yourself.
01Deal createdValue: $40,00002Value revised$40,000 → $48,00003Stage changedDiscovery → Proposal
Store what happened, derive the rest.
Two linked ideas. Event sourcing changes what you store. The unified log changes where everyone reads it from.
The everyday default is to store the current state and overwrite it. A field holds a value, a new edit replaces it, and the old value is gone. Event sourcing inverts that. You persist the sequence of immutable events that produced the state, facts about what happened, named in the past tense: an email arrived, a meeting was booked, a deal stage changed, a task was completed. Current state is then something you derive by folding over that sequence, as Fowler (2005) sets out.
The unified log is the companion idea. Put all of those events onto one ordered, append-only, durable stream that every system can publish to and read from. Kreps (2013) frames the log as the unifying abstraction underneath databases, replication, and stream processing, and the Kafka paper (2011) grounds it in a concrete, widely deployed commit log.
Two companions travel with these. CQRS separates the write path that appends events from the read path served by purpose-built views, an independent choice that is often paired with event sourcing. Change data capture, or CDC, turns a system's own change log, for example a CRM's record updates, into an event stream, so an existing store can feed the log without being rebuilt around it.
Event history
Current view
owner: Dana
stage: ProposalApply events in the defined order to compute the view.
The hard parts are real, and most of them are about agreement.
Getting events onto one log is the easy half. The difficulty is ordering them, knowing what is the same thing, and resolving genuine two-way conflicts.
Every source has its own clock.
Email, calendar, and two CRMs each have their own notion of "latest." A single total order is easy on one machine and genuinely hard across many. This is where logical clocks and per-source ordering matter, and where a naive "newest timestamp wins" quietly loses data.
One thing, many keys.
The same person, company, or deal appears in every source under a different key and a different spelling. Collapsing those into one entity, called entity resolution, without merging two real but distinct things is a core difficulty, and it sits upstream of everything else.
Both sides edit the same field.
When the connected view and a CRM can both change the same field, edits collide. Resolving them needs a defined merge policy, not a coin flip. CRDTs (Shapiro et al., 2011) are one principled answer: structures that merge concurrent edits deterministically so replicas converge without a central coordinator.
The same event, more than once.
Sources resend, retry, and deliver out of order. A consumer has to process the same event twice without double-counting, and to rebuild a view from scratch by replaying the log. Designing for that up front is what makes replay safe.
Events outlive their shape.
Events are kept for a long time, so the shape of an "email" or a "deal stage change" event will change. Old and new versions have to coexist and stay readable, which makes versioning a first-class concern rather than an afterthought.
Append-only meets "delete this."
An immutable log sits in tension with a request to remove a record. Handling deletion, retention, and access control over a durable event history is a real design constraint, and the field treats it as part of the design, not a patch.
Events in, derived views out, and a loop back.
Capture changes as events, keep them ordered, fold them into views, reconcile identity, and sync writes back through a conflict policy.
- 01Capture
Record domain events.
- 02Order
Preserve the required sequence.
- 03Project
Derive current views.
- 04Resolve
Link source identities.
- 05Synchronize
Apply conflict and loop policies.
The shape Kleppmann (2015) draws out: the log is the center, and views are derived, rebuildable things.
1Capture changes as events
Append, never overwrite.
Each source emits events, or CDC derives them from a system's own change log, and appends them to the stream. Events are immutable facts named in the past tense: EmailReceived, MeetingBooked, DealStageChanged, TaskCompleted. Nothing is edited in place, so the log keeps the full history of what happened rather than only the latest snapshot.
| Sequence | Event |
|---|---|
| 41 | EmailReceived |
| 42 | MeetingBooked |
| 43 | DealStageChanged: Proposal |
| 44 | TaskCompleted: send quote |
2Order and partition
Keep order where it matters.
The log keeps a stable order, and each record carries a position. To scale, it is partitioned, and ordering is guaranteed within a partition. Choosing a partition key, for example by account, keeps related events in order where it counts, which is what makes a derived view consistent for a given account even at volume. The Kafka paper (2011) describes this partitioned, durable design in full.
- Partition 0 · Acme
- 41 → 42 → 43 → 44
- Partition 1 · Globex
- 17 → 18 → 19
The streams can advance independently.
3Derive views by replay
Fold the stream into current state.
Consumers fold the event stream into materialized views. One view is the de-duplicated, current picture of the account. Different views can answer different questions, a timeline, an analytics rollup, all rebuildable from the same log. Because the log is the source of truth and views are derived, a view can be thrown away and rebuilt, or a brand new view added later over the full history.
DealCreated → MeetingBooked → StageChanged → TaskCompleted
- Account view
- Current owner, stage, and next step.
- Timeline
- The chronological activity history.
- Analytics
- Counts and time spent in each stage.
4Reconcile identity
Count one customer once.
An entity-resolution step maps source-specific keys to shared identities, so the derived view counts one customer once. This is the de-duplication that turns "acme@" in the inbox, "Acme Corp" in one CRM, and "ACME Inc." in another into a single record. The connection side of this, identity and link prediction, gets a fuller treatment in Knowledge and connection graphs.
| Source key | Canonical identity |
|---|---|
| CRM account 482 | Organization 1182 |
| Billing customer 91 | Organization 1182 |
| Email domain acme.com | Candidate for organization 1182 |
5Sync back, the two-way part
Writes are a loop, not a one-way street.
Writes from the connected view are themselves modeled as events or commands that flow back to the source systems. Because two systems can edit the same field, the return path runs through a defined conflict policy: last-writer-wins on a logical clock, a per-field merge, or CRDT-style convergence. That gate is what keeps the loop consistent rather than fighting itself.
- Receive a source change
Record source identity and a stable change identifier.
- Update the projection
Apply the event to the unified state.
- Write back if needed
Send the permitted change to another system.
- Recognize the echo
Avoid treating the same propagated change as a new independent edit.
6Replay and rebuild
The history is the safety net.
Because the log retains the full sequence, a view is disposable. You can drop it and replay from the start, fix a bug in how a view is computed and reprocess history, or add a new view later that reads the whole past. Kleppmann (2015) makes this the center of the model: the log is the system of record, and views are rebuildable projections over it.
- Keep the event history
Retain the source events and their versions.
- Create a new projection
Use the corrected transformation logic.
- Replay
Fold the retained events into the new view.
- Switch readers
Validate the result before using it.
One ordered stream becomes many views that agree.
At the field level: a single de-duplicated picture, provenance for every fact, decoupled tools, and the ability to ask what the past looked like.
Back to the salesperson from the top. Email, calendar, the CRMs, and the task list all read from and write to the same ordered record of what happened. The payoff the field describes is a single connected knowledge layer: one current view, kept field-level, where you can also answer where each fact came from. The hard cases above are exactly what decides whether that view stays trustworthy.
One current, de-duplicated view.
Assembled from email, calendar, multiple CRMs, and tasks, kept in step because every tool reads the same ordered stream and one customer is counted once.
Provenance and audit.
Every fact in the view traces back to the event that produced it, so you can answer "where did we learn this" and reconstruct any past state.
Decoupling.
Producers and consumers evolve independently. A new tool can subscribe to the log without the others changing, and CDC lets an existing store join without a rebuild.
Time travel and analytics.
Because history is retained, you can ask what the account looked like at any prior moment, and reprocess the past when the logic improves.
This page explains the field. It makes no claim about how any particular product, Prism included, is built, and stays neutral on the plumbing.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
The log as the system of record.
Data platforms that treat the log as the system of record rather than a buffer in front of a database, the "database inside-out" framing, with views derived on top.
Convergence on disconnected devices.
Collaboration built on CRDTs, pushing convergent two-way sync toward edits made offline, so devices reconcile cleanly when they reconnect.
Read the same log two ways.
Unified abstractions that let one log be read as a stream or as a queryable table, narrowing the gap between event sourcing and everyday querying.
The problems still being worked.
Privacy-respecting deletion over immutable logs, standardizing schema evolution, and making entity resolution reliable at scale remain active work.
The papers and essays behind this page.
- Kreps (2013). The Log: What every software engineer should know about real-time data's unifying abstraction. LinkedIn Engineering.
- Kreps, Narkhede, Rao (2011). Kafka: a Distributed Messaging System for Log Processing. NetDB 2011.
- Fowler (2005). Event Sourcing. martinfowler.com.
- Fowler (2011). CQRS. martinfowler.com.
- Shapiro, Preguiça, Baquero, Zawirski (2011). Conflict-free Replicated Data Types. SSS 2011.
- Kleppmann (2015). Turning the database inside-out with Apache Samza. martin.kleppmann.com.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
One ordered record of what happened, one view that agrees. See what a voice co-pilot feels like on a real call.
No app required. Just your phone number.