Distributed data processing at scale.
Scattered records across many systems go in. One current, de-duplicated view comes out. Here is the field that makes that happen, in plain language.
Distributed data processing is how you get a single, current, trustworthy answer out of data that lives in many places, keeps changing, and is too big for one machine. We walk it the way the field is built: what it is, why it is hard, how the named systems do it, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Many machines, working together on one answer.
When work is too big, too scattered, or too continuous for one machine, the job is split across many machines that run in parallel, and a framework presents one result.
A relatable version of the problem: picture a salesperson whose working life is spread across email, a calendar, one or more CRMs, and a couple of task lists. The same customer shows up in all of them, under slightly different names, with slightly different details, and each system is edited independently through the day. Pulling all of that into one current, de-duplicated view, a connected knowledge layer, is a distributed data processing problem.
Two broad shapes of work cover most of it. Batch processing rebuilds a large fixed dataset, for example the full view overnight. Streaming folds in each new event as it arrives, for example each new email or CRM edit. Modern frameworks aim to express both with one model. Underneath, a framework always does three jobs: partition the work and data, shuffle records between machines so related ones meet, and recover from the failures that are normal at scale. A data pipeline is the path data travels from sources, through cleaning, joining, and de-duplication, to a usable view. Sync keeps two or more copies agreeing over time. Two-way sync lets changes flow in both directions, which adds the question of how to reconcile concurrent edits.
- Partition
Split a large dataset into smaller pieces.
- Process
Workers clean, transform, or join their partitions.
- Combine
Group partial results by key.
- Recover
Retry failed work using the framework’s recorded state.
The data is large, scattered, and always changing at once.
Scattered data is not a tidy table you read once. It is large in aggregate, it arrives continuously and out of order, the same real-world thing appears many times, and edits happen in more than one place at once.
Failure is the normal case.
With enough machines, something is always failing. Correctness has to survive machines dying mid-job, so frameworks build in re-execution and recovery rather than assuming success.
Moving data is expensive.
The shuffle, sending records across the network so related ones meet, is often the costliest part of a job. Much of the field is about computing more and moving less.
The same thing appears many times.
Across email, a calendar, several CRMs, and task lists, one customer shows up under many slightly different records. Comparing every record to every other is quadratic, so systems use blocking and indexing to compare only plausible candidates. Entity resolution is covered in depth in Knowledge graphs.
Data arrives out of order.
A late or delayed event can change an answer you already emitted. Deciding when an answer is final, and how to revise it when late data lands, is a core difficulty named directly by the Dataflow Model.
Two copies change at once.
In two-way sync, when two copies change the same field independently, there is no single clock to say which is right. Resolving this without a central referee, so all copies eventually agree, is what conflict-free replicated data types address.
A correct view drifts.
A view that was correct an hour ago drifts as sources change. The pipeline has to keep folding in change without reprocessing everything from scratch.
Six named ideas turn scattered data into one current view.
Split the work, keep it in memory, query it fast, log every change, fold in late data, and let independent copies converge. Each idea is anchored to its paper.
- 01Batch
Parallel map and reduce.
- 02Memory
Reuse intermediates and lineage.
- 03Query
Read selected columns.
- 04Log
Retain ordered changes.
- 05Time
Handle windows and late events.
- 06Merge
Use defined convergence rules.
The lineage this page follows: six building blocks, each doing one job, each with a source paper.
1Batch processing with a simple parallel model
Split the work, combine by key.
Express a job as a map step, which transforms each record independently, and a reduce step, which combines records that share a key. The framework parallelizes across a cluster, handles machine failures by re-running work, and manages the shuffle. The lasting lesson is the programming model: hide distribution and fault tolerance behind two simple functions, in MapReduce (Dean & Ghemawat, 2004).
Map workers
| Worker | Partial count |
|---|---|
| 1 | Acme: 2 |
| 2 | Acme: 3 |
| 3 | Globex: 1 |
Reduce by key
2In-memory, multi-pass processing
Keep data in memory between passes.
Many useful jobs touch the same data many times: joins, iterative cleanup, de-duplication passes. Keeping data in memory between passes, and rebuilding any lost piece from its recorded lineage rather than by replication, makes multi-pass and iterative work far more efficient than writing to disk between every step. That is the resilient distributed dataset abstraction behind Spark, in Zaharia et al. (2012).
- Load partition 2
Read its source slice.
- Filter and transform
Record the operations that produce the cached partition.
- Worker failure
The cached copy is lost.
- Recompute
Replay the required lineage to regenerate that partition.
3Interactive query over huge data
Read only what a question needs.
A columnar layout reads only the fields a query touches, and a multi-level serving tree fans a query out across many machines and aggregates the partial answers back up. Together they answer ad-hoc questions over very large nested datasets in seconds instead of minutes. This is the lineage behind interactive "ask a question of everything" experiences, in Dremel (Melnik et al., 2010).
Count won deals by owner.
| Column | Read? |
|---|---|
| owner | Yes: grouping key |
| stage | Yes: filter |
| description | No |
| attachment | No |
Workers aggregate locally; the serving tree combines the totals.
4The change log as the pipeline backbone
One ordered stream of changes.
An append-only, ordered log of events sits between sources and consumers. Producers append changes, many consumers read at their own pace, and the ordered log gives every downstream system the same sequence of events to build from. This is the common spine for change data capture: turn every edit in a source system into an event others can react to, as described in Kafka (Kreps, Narkhede, Rao, 2011).
| Position | Event |
|---|---|
| 41 | Email received |
| 42 | Stage changed |
| 43 | Meeting booked |
| 44 | Task created |
- View A
- Applied through 44.
- View B
- Applied through 42; will resume at 43.
5One model for batch and streaming
Fold in late data, revise cleanly.
Separate four plain questions: what is computed, where in event time results are grouped, called windowing, when results are emitted, governed by triggers and watermarks, and how later data refines earlier answers. This is how a system keeps a current view as events arrive late and out of order, without choosing batch correctness or streaming freshness over the other, in the Dataflow Model (Akidau et al., 2015).
6Reconciling concurrent edits without a referee
Independent copies that always converge.
Conflict-free replicated data types are data structures designed so that independent, concurrent updates on different copies can always be merged into the same final state, with no central coordinator. Replicas may differ for a moment but are guaranteed to converge. This is the principled answer to the two-way-sync conflict question, what happens when the view and a source system both change the same thing, in Shapiro, Preguica, Baquero, Zawirski (2011).
Phone edit
stage: ProposalLaptop edit
next_step: Send quote Fridaystage: Proposal · next_step: Send quote Friday
De-duplication, deciding which records are the same real-world thing, is the natural seventh idea here. It is owned in depth by Knowledge graphs, so this page names it as a pipeline stage and links across rather than repeating the depth.
Scattered, changing data becomes one view you can trust.
At the field level: one current, de-duplicated view, answers in seconds over large data, freshness without giving up correctness, two-way sync that survives concurrent edits, and resilience as a property of the system.
A seller's world is a sync-and-de-dup problem. The same customer lives in email, a calendar, one or more CRMs, and task lists, under slightly different names, and every system is edited independently through the day. The outcomes this field points at are exactly the outcomes a strong co-pilot delivers. We frame the bridge at the level of what a reader gets, never how anything is built.
One de-duplicated view per account.
A single, current record of each account and contact, so it is trustworthy and a rep is never reconciling systems by hand. See Clean CRM.
A connected knowledge layer.
Reading across the scattered sources at once, instead of leaving them siloed, so context is in one place. See Deep context.
A view that stays current.
As new email, meetings, and edits land, the record keeps up without the rep doing the stitching by hand.
Two-way sync that survives concurrent edits.
When the same record is changed in two places at once, those edits are reconciled into one view rather than one silently overwriting the other, so what a rep sees stays trustworthy.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Batch and streaming as one.
The same pipeline serves both the overnight rebuild and the live update. The Dataflow Model line of work pushed this into the mainstream.
The change log as source of truth.
Architectures where an ordered stream of changes is the source of truth, and many views are derived from it.
Work offline, merge cleanly.
Conflict-free replicated data types let people work on independent copies, a phone, a laptop, and merge cleanly when they reconnect.
Update as inputs change.
Incremental and materialized views that fold in new change as it lands, instead of reprocessing everything from scratch.
Permissions travel with the data.
Stronger data governance, where permissions, lineage, and isolation travel with the data rather than being bolted on afterward.
The papers behind this page.
- Dean & Ghemawat (2004). MapReduce: Simplified Data Processing on Large Clusters. OSDI 2004.
- Zaharia et al. (2012). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. NSDI 2012.
- Melnik et al. (2010). Dremel: Interactive Analysis of Web-Scale Datasets. VLDB 2010.
- Kreps, Narkhede, Rao (2011). Kafka: a Distributed Messaging System for Log Processing. NetDB 2011.
- Akidau et al. (2015). The Dataflow Model. VLDB 2015.
- Shapiro, Preguica, Baquero, Zawirski (2011). Conflict-free Replicated Data Types. SSS 2011. (Open-access tech report: hal.inria.fr.)
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
When your systems stay this clean and this current on their own, the admin stops being your job. See it on a real call.
No app required. Just your phone number.