Stream processing: data as it happens.
A constant flow of changes goes in. One current, de-duplicated view comes out. Here is the field that makes that happen, in plain language.
Stream processing treats data not as a thing you fetch, but as a never-ending sequence of events that arrive over time, and it reacts to each one as it happens. We walk it the way the field is built: what it is, why keeping things current is hard, how a modern system does it, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
A late event may revise the window’s result.
Data as an unending flow of events.
Most software treats data as a thing you fetch. Stream processing treats it as a flow you react to, and keeps the answer current.
A stream is an unbounded, ever-growing sequence of events, also called records or messages. An event is a small fact with a time attached: this changed, at this moment. Contrast that with batch processing, the model most people already know: collect a bounded pile of data, run a job over all of it, produce an answer, stop. Streaming never stops. The data keeps coming, and the answer is continuously kept current.
Here is the relatable version of the problem. A salesperson lives across an inbox, a calendar, one or more CRMs, and a task list. The same person, company, and deal show up in all of them, under slightly different names, at slightly different times, sometimes edited in two places at once. A CRM update, a moved meeting, a new email, a completed task are all events. Keeping all of that as one current, de-duplicated view, and keeping it two-way so a change in any system shows up correctly in the others, is a stream processing problem in disguise. The value is that the picture is current, not that it is large. The anchor phrase for the whole field: data as it happens.
Batch
- Bounded input
Read a fixed dataset.
- Job completes
Return its result.
Stream
- Unbounded input
Events continue arriving.
- State updates
Maintain a result as new events arrive.
The moment you want it up to date, the easy assumptions break.
Events arrive late, out of order, and sometimes twice. The same record gets touched in two places before either change propagates.
When it happened is not when you see it.
A phone goes offline, then syncs an hour of changes at once. Event time, when something happened, differs from processing time, when your system sees it. This is one of the most important distinctions in the field, and the source of most of the subtle bugs.
Events arrive out of order.
Events do not arrive in the order they occurred, and some arrive late. A processor that assumes order computes the wrong answer. You need a principled way to decide whether you have probably seen everything up to a given time, which is what a watermark is.
The same change can land twice.
Networks retry. A delivery you thought failed actually went through. Without care, the same change gets applied twice: a deal logged twice, a task created twice. De-duplication and exactly-once semantics exist to make applying a change safe to repeat.
Two truths for one field.
When the same record is edited in two systems before either change propagates, you get a conflict. Resolving it needs a policy: last-writer-wins by timestamp, per-field merge, or a data type that merges deterministically. Naive two-way sync creates update loops and silent overwrites.
The processor has to remember.
To de-duplicate, and to know what changed, the system must remember things: which events it has seen, the current value of each record. That state must survive crashes and scale across machines, which is hard to get right without losing or double-counting it.
You need a record you can replay.
A durable, ordered, replayable log of changes is what lets a system rebuild its state, recover from failure, and add a new consumer later without losing history. Without it, a crash or a new reader means lost ground.
The field turns a messy flow into one current view.
Six mechanisms, each introduced as the fix for a concrete failure above. This is the earned-depth core, so the real names are here.
1The event log
Durable, ordered, replayable.
The backbone of modern streaming is a log. Changes are appended to an ordered, persistent record that many independent consumers can read at their own pace, and replay from the beginning if needed. Kafka (Kreps, Narkhede, Rao, 2011) is the seminal system here. For the sync example, this is the idea that every change becomes a durable event other systems can subscribe to. Change data capture, or CDC, is the common pattern for turning edits in a database or app into such an event stream.
| Position | Event |
|---|---|
| 0 | Email received |
| 1 | Meeting booked |
| 2 | Deal updated |
| 3 | Task created |
- Reader A
- Applied through position 3.
- Reader B
- Applied through position 1; resumes at 2.
2Event time, watermarks, windowing, triggers
Reason about time correctly.
The Dataflow Model (Akidau et al., 2015) separated four questions every streaming computation answers: what is computed, where in event time the result belongs (which window), when in processing time results are emitted (triggers, driven by watermarks), and how later refinements relate to earlier results (accumulation). A watermark is the system's estimate that it has probably seen every event up to a given time. This is the conceptual spine for handling late and out-of-order data without computing the wrong answer.
| Arrival order | Event time | Event |
|---|---|---|
| 1 | 10:01 | A |
| 2 | 10:03 | C |
| 3 | 10:04 | D |
| 4 | 10:02 | B arrives late |
B belongs to the earlier time window despite arriving last.
3Stateful processing with fault tolerance
Remember without losing or double-counting.
Real processors keep state, such as running values and seen-event sets, and must not lose or double-count it on failure. Apache Flink (Carbone et al., 2015) popularized consistent state snapshots, so a job can recover to an exact point. Discretized Streams (Zaharia et al., 2013), behind Spark Streaming, took a different route: cut the stream into tiny deterministic batches and use lineage to recover. Two honest, contrasting designs worth naming. Any performance numbers reported in those papers belong to their authors and their setups, not to anyone reading this page.
- Running state
Processed through event 43.
- Checkpoint
Save the state with its corresponding input positions.
- Failure
Event 44 is interrupted.
- Restore
Reload the checkpoint and replay subsequent input.
4Exactly-once and de-duplication
The same change lands once.
Combine idempotent writes, unique event keys, and consistent state checkpoints, and replays or retries stop double-applying a change. This is what makes the same deal not getting logged twice a property of the system rather than a hope. In the sync example, it is the difference between one clean contact and the same person showing up two or three times.
| Delivery | Sink decision |
|---|---|
| deal-update:7c2 | Apply and record key |
| deal-update:7c2 again | Recognize duplicate |
| deal-update:7c3 | Apply new change |
5Two-way sync and conflict resolution
Two edits, one record, a clear policy.
Keeping multiple systems consistent means mapping records to a stable identity, so the same person is one entity and not three, propagating changes both ways without loops, and resolving simultaneous edits with a stated policy. Approaches range from last-writer-wins, to per-field merge rules, to CRDTs, conflict-free replicated data types, that merge deterministically by construction. Conflict resolution is a policy choice with trade-offs, not a single solved answer.
CRM A
stage: ProposalCRM B
owner: Danastage: Proposal · owner: Dana
6The architecture choice: Lambda versus Kappa
Two paths or one.
Lambda runs two paths: a batch layer for accurate history and a speed layer for fresh-but-approximate results, then merges them. Kreps (2014) argues that a single streaming path over a replayable log can serve both, by replaying history through the same code, an approach since called Kappa. Present it as an honest trade-off: Lambda buys a safety net at the cost of two code paths to keep in sync, Kappa buys simplicity at the cost of leaning hard on the log and on streaming maturity. This is a general comparison in the field, not a claim about any specific system.
Lambda-style
- Batch path
Recompute from historical data.
- Speed path
Produce fresh incremental results.
- Serving layer
Combine the results.
Kappa-style
- Replayable log
Retain source events.
- Stream processor
Use the same processing path for live input and replay.
A clean, current, shared view of what is true now.
At the field level: a current view instead of a stale one, one record per real thing, two-way consistency, replay and recovery, and the ability to react in the moment.
Back to the salesperson living across an inbox, a calendar, one or more CRMs, and a task list. When changes are events on a durable log, processed with exactly-once care and a stated conflict policy, the messy flow becomes a connected layer that is right now. The hard cases in the section above are exactly the ones that decide whether that view can be trusted.
A current view, not a stale one.
The picture reflects what is true now, across all the connected systems, instead of whatever was loaded last.
One de-duplicated record per real thing.
Because changes carry identity and are processed with exactly-once care, the same contact or deal stops showing up two or three times.
Two-way consistency.
A change made in one place shows up correctly in the others, with conflicts handled by a stated policy rather than silent overwrites.
Replay, recovery, and reacting in the moment.
A durable log lets a new consumer catch up from history and the system rebuild after a failure, and it lets the system act on each change as it happens rather than on a nightly schedule.
This page describes the field, not our plumbing. We stay neutral on how any specific product implements sync, de-duplication, or conflict resolution.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Streaming as the default.
The Dataflow lineage treats a bounded batch as simply a stream that ends, unifying the two models under one set of concepts, with batch as the special case rather than the starting point.
Streaming databases and live materialized views.
Query results maintained incrementally as events arrive, blurring the line between a database and a stream processor, so a view stays current without a re-run.
Exactly-once and state, made simpler.
Continued work to make correct, fault-tolerant, stateful streaming the easy default rather than the expert path.
Local-first and CRDT-based merging.
Conflict-free data types and replayable logs pushing toward sync that tolerates offline edits and merges them predictably, without silent overwrites.
Schema and identity in the pipeline.
More of the hard work, resolving which record is which and validating shape, handled as part of the pipeline, so downstream views stay clean.
The papers behind this page.
- Akidau, Bradshaw, Chambers, Chernyak, Fernandez-Moctezuma, Lax, McVeety, Mills, Perry, Schmidt, Whittle (2015). The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. Proceedings of the VLDB Endowment.
- Kreps, Narkhede, Rao (2011). Kafka: a Distributed Messaging System for Log Processing. NetDB workshop.
- Carbone, Katsifodimos, Ewen, Markl, Haridi, Tzoumas (2015). Apache Flink: Stream and Batch Processing in a Single Engine. IEEE Data Engineering Bulletin.
- Zaharia, Das, Li, Hunter, Shenker, Stoica (2013). Discretized Streams: Fault-Tolerant Streaming Computation at Scale. SOSP.
- Kreps (2014). Questioning the Lambda Architecture. O'Reilly Radar.
Keep reading.
Sibling explainers in the Research section, same Pipelines and sync theme.
Talk. Done.
One current, de-duplicated view of your work, kept right now. You talk, and the picture stays in sync. See it on a real call.
No app required. Just your phone number.