Data lakes and the lakehouse.
Many partial copies of the same record go in. One current, de-duplicated view comes out, and a correction made once flows back. Here is the field that makes that happen, in plain language.
A data lake pools raw data from many systems into one low-cost store. A lakehouse adds a thin layer of bookkeeping so that pile of files behaves like a real, reliable table. We walk it the way the field is built: what it is, why it is hard, how the field does it, what it unlocks, and where it is heading. The papers and specs behind each idea are linked, so you can read the source yourself.
One pool of raw data, plus a layer that makes it trustworthy.
A data lake answers "where does everything live." A lakehouse answers "and can I trust a single record in it."
Picture one salesperson's working day spread across five places at once: email, a calendar, two CRMs (the company standard plus the one a past team never migrated off), and a task list. The same customer lives in all of them. The email spells the name one way, the first CRM has an old job title, the second CRM has a phone number the first one is missing, and the calendar knows about a meeting nobody logged. Each tool holds a piece of the truth, and no single tool holds the whole of it.
A data lake is the idea of pouring all of that raw material into one large, low-cost store, usually cloud object storage, in open file formats, instead of leaving it locked inside each separate app. That solves "where does everything live." But a lake on its own is just a pile of files. It has no agreed notion of a record, no safe way to update one fact, and no protection when two writers touch the same thing at once.
A table format is the bookkeeping layer that sits over those files and tells an engine which files make up a table right now, what the schema is, and what changed. That is what turns a folder of files into a table you can trust. Delta Lake, Apache Iceberg, and Apache Hudi are three widely used open examples. The lakehouse is the resulting architecture: cheap open storage plus a transactional table layer, so one system serves both raw analytics and the reliable, updatable records an application reads from.
- Data files
Columnar files live in object storage.
- Table metadata
A commit identifies the files and schema belonging to a version.
- Reader
Reads a consistent snapshot rather than an arbitrary folder listing.
Cheap storage is not a database, and the truth is scattered.
Pooling the data is the easy part. Making one record correct, current, and safe to change across many systems is where the real problems live.
Cheap blobs, weak guarantees.
Cloud object storage holds keys and blobs cheaply, but listing files is slow and costly and there is no built-in transaction across multiple files. Getting database-grade reliability on top of that is the core problem Delta Lake set out to solve.
No half-applied changes.
Updating one customer can touch several files. A reader must never catch a half-written state, and two concurrent writers must not corrupt each other. That needs an explicit transaction log, not file overwrites.
Three records, one human.
The same person shows up under slightly different spellings, stale titles, and partial fields. Deciding that those records are one entity, and merging them without dropping good data, is a probabilistic matching problem with a formal theory behind it, not an ad-hoc string compare.
Corrections have to flow back.
A connected view that only reads is half a tool. Pushing a fix back to the source systems raises the classic question: when two copies diverge, which write wins, and can the systems converge without a central referee.
Current, without rescanning everything.
"Current" means absorbing a steady stream of changes and applying them quickly. Incremental upserts and incremental reads are the mechanisms, so the view stays fresh as email, calendar, and CRM data keep moving, without rewriting the whole table.
Sources change shape over time.
Systems add and rename fields as they evolve. The table layer has to absorb that safely, without rewriting history or breaking readers that expect the old shape.
Cheap storage on the floor, a log of changes on top.
Treat the stream of changes as the source of truth and every view as derived from it. That makes both de-duplication and two-way flow easier to reason about.
- 01Store
Keep files in object storage.
- 02Track
Record table commits.
- 03Plan
Select files through metadata.
- 04Update
Apply changes to records.
- 05Resolve
Link duplicate entities.
- 06Reconcile
Handle edits across systems.
Six jobs, each doing one thing: from a pile of files to a current, de-duplicated, connected record.
1The floor and the log
Open files, with a log over them.
Open columnar file formats, for example Parquet, hold the data cheaply and stay readable by many engines, which avoids lock-in. On its own that is just files. The table layer keeps an ordered log of what each commit added or removed, and a reader computes the current table from the log.
That single move buys a lot: atomic commits, snapshot isolation so a reader never sees a half-written state, and time travel, reading the table as of an earlier version. Time travel is also how you undo a bad merge. The canonical description of getting these guarantees over cloud object storage is Delta Lake (Armbrust et al., 2020).
| Version | Change | State |
|---|---|---|
| v1 | Initial load | Original records |
| v2 | Verified correction | Known good state |
| v3 | Incorrect merge | Needs investigation |
Possible while the required history and data files are retained.
2Metadata that scales
Plan from metadata, not by listing files.
As a table grows to millions of files, listing them to answer a query becomes the bottleneck. The Apache Iceberg table spec tracks files through layered manifests, so an engine can plan a query over a very large table from metadata alone. The same spec covers schema evolution and snapshot isolation, which is how the table stays both queryable and safe to change as sources drift.
- Snapshot
Identifies the current table version.
- Manifest list
Finds the manifests belonging to that snapshot.
- Manifests
Describe data files and statistics.
- Selected files
Read only files relevant to the query.
3Upserts and incremental processing
Apply only what changed.
Rather than rewrite a whole table, the system merges in only the changed records, an upsert, and lets downstream consumers pull just what changed since a known point. This is how a unified view stays current as email, calendar, and CRM data keep moving. Apache Hudi is a table platform built around exactly this: upserts and incremental reads, designed to absorb a steady stream of changes without full rewrites.
Update the affected records using the table’s write strategy.
4De-duplication and entity resolution
Decide that three records are one person.
Candidate records are compared field by field and scored for the probability that they refer to the same entity, then linked or merged above a threshold, with uncertain pairs flagged for review. The Fellegi-Sunter model (1969) gives the probability framework for that decision. The point worth holding: de-duplication is a measured decision, not a guess, and the uncertain cases are routed to a human rather than merged blindly.
| Candidate | Evidence |
|---|---|
| Jordan Avery | Verified email: [email protected] |
| J. Avery | Same verified email |
| Jordan A. | No verified identifier |
Keep the third candidate separate until there is enough evidence.
5Two-way sync and conflict resolution
When both sides change, both sides converge.
For corrections to flow back, concurrent edits from different sources have to merge deterministically, or an explicit, auditable policy has to decide the winner. CRDTs (Shapiro et al., 2011) are one principled answer: data types designed so that concurrent edits on different copies merge to the same state, so every replica converges without a central referee. The honest alternative is a stated policy, last writer wins, source priority, or human review, applied where automatic convergence does not fit.
CRM A
title: VP OperationsCRM B
phone: 555-0148title: VP Operations · phone: 555-0148
A current, de-duplicated, connected view of the work.
At the field level: one trustworthy record assembled from many partial copies, corrections that propagate, a history you can audit, and one store serving many uses.
For the sales example, the payoff is a connected knowledge layer: the same customer, assembled from email, calendar, both CRMs, and the task list into one record that stays fresh and field-level correct. This section describes what the field makes possible. It does not describe how any one product builds it.
One current, de-duplicated view.
A single contact, deal, or account assembled from many partial copies, kept fresh as the sources change, with the good fields from each kept and the duplicates resolved.
Corrections that propagate.
A fix made once can flow back to the systems that need it, instead of being re-entered tool by tool. The hard part, deciding which change wins, is the two-way sync problem from above.
Auditability and undo.
Because the table layer keeps versioned history, you can see what a record looked like last week and roll back a bad merge, rather than guessing what changed.
One store, many uses.
The same reliable layer serves reporting, search, and the structured records an assistant or application reads from, without copying data into yet another silo.
Everything on this page describes the field, not our plumbing. We make no claim about how AI REV or Prism stores, syncs, or de-duplicates anything.
The direction the field is heading.
We describe the direction, not a roadmap of our own, and we stay honest about what is still unsolved.
Not trapped in one table format.
The widely used table formats are moving toward interoperability, so data is not locked into one of them. Open catalog standards are an active area.
Closer to real time.
Tighter coupling of change capture and incremental tables pushes "current" closer to the moment a fact changes, rather than a periodic batch.
Matching that learns.
Learned matching and embedding-based candidate generation aim to raise de-duplication accuracy on messy, multilingual, real-world records.
Honest about what is unsolved.
Conflict-resolution policy for two-way sync, governance and access control over a shared layer, and keeping metadata fast as tables grow are all active research, not solved.
The papers and specs behind this page.
Iceberg and Hudi are project specs and platforms rather than single papers, and appear here as honest references to public, open-source work.
- Armbrust et al. (2020). Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. PVLDB 13(12).
- Zaharia, Ghodsi, Xin, Armbrust (2021). Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. CIDR 2021. (Author mirror: eecs.berkeley.edu.)
- Apache Software Foundation. Apache Iceberg Table Spec (originated at Netflix). Project specification.
- Apache Software Foundation. Apache Hudi (originated at Uber). Open-source table platform.
- Shapiro, Preguica, Baquero, Zawirski (2011). A Comprehensive Study of Convergent and Commutative Replicated Data Types. INRIA RR-7506.
- Fellegi and Sunter (1969). A Theory for Record Linkage. Journal of the American Statistical Association, 64(328).
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
One scattered customer becomes one current record you can just talk to. See it on a real call.
No app required. Just your phone number.