Cleaning data with minimal human effort.
A large dataset is mostly right and dirty in places. A person can read only a sliver of it. Here is the field that decides which sliver.
Every record set drifts over time: the account entered twice, the contact with a missing title, the deal that went quiet months ago. We walk the field the way it is built: what it is, why it is hard, how a modern system keeps a record set clean while asking a person about very little, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
- 01Clear agreementLower priority
- 02Ambiguous matchAsk a person
- 03Clear differenceLower priority
Reviewer: Different companies
Get a messy dataset into good shape, by reading almost none of it.
The scarce resource is human judgment. The field is about spending it only where it counts.
Take a dataset that is large, mostly correct, and dirty in places: duplicates, missing fields, inconsistent values, stale entries. Produce a cleaner version. The constraint is that a person can review only a tiny fraction of the rows. The research question is which fraction. A duplicate that is obvious can be merged on its own. A name that clearly matches can be linked on its own. The expensive step is the human call, so the goal is to spend it only where the machine is genuinely unsure, or where one decision fixes many rows.
Three recurring sub-problems show up. Deduplication, also called entity resolution, decides when two records describe the same real-world thing: the account that shows up twice. Repair and imputation fix or fill wrong and missing values: the half-filled contact. Validity and freshness flag records that break the rules of the data or have gone quiet: the dead pipeline deal. Two toolkits do most of the work, and real systems combine them. Active learning makes a person's answers go further by choosing the most informative thing to ask. Inference-based cleaning uses rules and statistics to repair the easy cases automatically and surface only the hard ones.
- Detect
Find possible duplicates, missing values, and rule violations.
- Select
Choose cases whose labels would help the model most.
- Review
A person checks the evidence and supplies a label.
- Update
Learn from the label, then evaluate on held-out records.
The value hides in a small, ambiguous minority.
Most records are easy. The difficulty is finding the few that matter, before a person looks at any of them.
Random sampling wastes attention.
Picking records to review at random spends a person's time on obvious cases. The hard part is locating the small set of ambiguous records where a decision actually changes the outcome, before anyone reads a row.
Matching is quadratic and noisy.
Comparing every record to every other does not scale: a million records make roughly half a trillion pairs. So systems block, grouping plausible matches cheaply first. The comparison itself is fuzzy: "Acme Inc." against "ACME Incorporated", nicknames, typos, a moved address.
Fixing one field moves another.
Repairing a value can change whether two records match. Merging two records can expose a fresh conflict. Cleaning is not a tidy row-by-row pass, which is why some approaches reason over the whole dataset and many signals at once.
Cleaning changes the answer.
When a dataset feeds a model or a report, cleaning some rows and not others can bias the result. Doing it without distorting what the cleaned data is used for is its own research problem.
Hand-labeling does not scale.
Building enough labeled examples to train a matcher is slow and costly, and the moment the data drifts, the labels go stale. That pressure is what motivates weak supervision: programmatic, reusable sources of labels instead of one-by-one annotation.
A wrong auto-merge is costly.
No automatic cleaner is perfect. "Clean" is a threshold the team sets, not a guarantee, and folding two real companies into one is expensive to undo. Good systems are tuned to ask a person when a mistake would hurt.
Automate the confident decisions, escalate the ambiguous ones.
Block candidates, resolve the easy cases with rules and statistics, learn where labels are scarce, and spend human questions on the genuine remainder.
- 01Block
Find plausible candidate pairs.
- 02Resolve
Apply validated rules and models.
- 03Learn
Use labels to improve decisions.
- 04Review
Ask a person about informative cases.
A practical pipeline: four jobs in a row, with a person seeing the few percent that genuinely need a human call.
1Active learning, and uncertainty sampling
Let the model pick its own questions.
Start with a small labeled set and a large unlabeled pool. Train a model, score every unlabeled record by how informative its label would be, ask a person about the top few, add their answers, retrain, and repeat. Active learning names the human the oracle and the question a query. The design choices are what to ask, how many at once, and how to keep each answer cheap.
The classic scoring rule is uncertainty sampling from Lewis & Gale (1994): ask about the records the model is least sure of. In a yes-or-no match decision, those are the pairs sitting nearest the 0.5 line. For the wider menu of scenarios and strategies, including query-by-committee and expected error reduction, the widely cited map is the Settles survey (2009).
Illustrative match probabilities. Review selection also depends on coverage and the cost of a wrong merge.
2Entity resolution, as a learned decision
Block, then compare.
For deduplication, a system first blocks: it cheaply groups records that could plausibly match, by shared email domain, ZIP, or a name key, so it never has to compare every record to every other. Inside each block it builds a feature comparison for each pair, things like string similarity, a matching tax ID, a shared domain, and learns a match or no-match classifier.
Modern versions compare attributes with deep learning. The DeepMatcher design-space study (Mudgal et al., 2018) maps where those models help and where simpler methods are enough. Active learning sits on top: label only the pairs the matcher is unsure about, so the labeled set stays small.
| Blocking key | Records to compare |
|---|---|
| Domain: acme.com | Acme Corp · ACME Corporation |
| Postal: 94107 | Records at nearby addresses |
| Name prefix: smi | Smith · Smithson |
3Cleaning as probabilistic inference
A repair is a best guess with a confidence.
Instead of hand-written fix-it scripts, describe what valid data looks like: integrity constraints like "ZIP determines city", reference tables, and the data's own statistics, such as a column that is almost always a five-digit ZIP. Then let the system infer the most probable repair for each error, and attach a confidence to it. This is the approach in HoloClean (Rekatsinas et al., 2017).
The confidence is what makes it efficient. High-confidence repairs apply on their own. Low-confidence ones are exactly the cases to route to a person. Because the repair comes with a reason, a rule or a reference it leaned on, the fix is inspectable rather than a black-box overwrite, which matters when the data drives decisions people answer for.
City: San Jose · ZIP: 94107
- Evidence
- The reference postal table maps 94107 to San Francisco.
- Open question
- Is the city wrong, or was the ZIP mistyped?
Change a field only after checking which value is supported.
4Clean what moves the result
Spend effort where it changes the outcome.
When cleaned data feeds a model, not every dirty row matters equally. ActiveClean (Krishnan et al., 2016) prioritizes cleaning the records most likely to change the model's behavior, and retrains correctly as the cleaning happens, so a small, well-chosen effort approximates the result of having cleaned everything.
In CRM terms, as an example of the idea and not a product claim: if duplicate accounts are skewing a forecast, the records worth cleaning first are the ones the forecast actually leans on, not every near-copy in the table.
- Round 1
A person labels a difficult duplicate pair.
- Round 2
The model uses that label to score similar candidates.
- Check
Measure false merges and missed duplicates on held-out data.
- Repeat
Select the next informative cases.
5Weak supervision, instead of one-by-one labels
Write rules, not annotations.
Rather than labeling examples one at a time, write labeling functions: small, noisy, programmatic rules. "If the email domains match, lean same company." "If the tax IDs differ, lean not a match." Each rule is imperfect on its own. A model estimates how reliable each one is and combines them into probabilistic training labels, replacing much of the hand-labeling. This is the method in Snorkel (Ratner et al., 2017).
It is how a team can bootstrap a matcher or a field-validator without a large annotated set, and reuse the rules as the data grows.
| Rule | Vote |
|---|---|
| Matching verified email | Match |
| Conflicting tax identifiers | Non-match |
| Both identifiers missing | Abstain |
Combine noisy, potentially dependent votes into a probabilistic training label.
A large, living dataset a small team can keep trustworthy.
At the field level: fewer duplicates, more complete records, fresher state, and a human in the loop for judgment rather than for volume.
For anyone evaluating a system that touches CRM data, the reassurance is simple: keeping a large record set trustworthy is a known, studied discipline with clear methods and honest trade-offs. A CRM full of duplicate accounts, half-filled contacts, and stale deals quietly poisons forecasts, routing, and reporting, and the field above is exactly how that is kept in check at scale. We point to the methods. We do not make claims here about how any particular product does it.
One clean account, not three near-copies.
Entity resolution is how a record set ends up with a single account instead of a scatter of duplicates, with the obvious merges handled and the doubtful ones escalated.
Gaps filled from consistent signals.
Inference-based repair fills missing fields from rules, reference data, and statistics, each fix carrying a confidence so the uncertain ones reach a person.
Effort that compounds.
Active learning and weak supervision mean each human decision teaches a model that handles the next thousand similar cases, so work spent early keeps paying off.
Auditable, not a black box.
Labeling functions and inference-based repair make why a record was merged or fixed inspectable, which matters when the data drives decisions people are accountable for.
For how AI REV approaches data and records specifically, see Clean CRM and Security. Everything on this page describes the field, not our plumbing.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Models as matchers and repairers.
Large language models can compare records, propose fills, and explain a match in zero- or few-shot settings, shrinking the labeled data a matcher needs. Active learning is being used to pick the few examples worth labeling or verifying.
Low-resource cleaning.
Reusing a matcher trained on one domain to bootstrap another, so a new dataset does not start from zero labels.
Data quality as engineering.
A broad shift from tuning models toward systematically improving data, with cleaning, labeling, and selection treated as measurable engineering rather than a one-off chore.
Humans for judgment, machines for scale.
The durable direction is a clean split: automate the confident decisions, escalate the ambiguous ones, and make each escalation cheap and clear.
Confidence with candor.
No automatic cleaner is perfect. A wrong auto-merge is costly, "clean" is a threshold and not a guarantee, and good systems are tuned to ask a person when the cost of a mistake is high.
The papers behind this page.
- Lewis & Gale (1994). A Sequential Algorithm for Training Text Classifiers (uncertainty sampling). ACM SIGIR. arXiv:cmp-lg/9407020.
- Settles (2009). Active Learning Literature Survey. Univ. of Wisconsin-Madison, Computer Sciences Technical Report 1648.
- Rekatsinas, Chu, Ilyas & Ré (2017). HoloClean: Holistic Data Repairs with Probabilistic Inference. VLDB. arXiv:1702.00820.
- Krishnan, Wang, Wu, Franklin & Goldberg (2016). ActiveClean: Interactive Data Cleaning While Learning Convex Loss Models. VLDB. arXiv:1601.03797.
- Mudgal, Li, Rekatsinas, Doan et al. (2018). Deep Learning for Entity Matching: A Design Space Exploration. ACM SIGMOD. doi:10.1145/3183713.3196926.
- Ratner, Bach, Ehrenberg, Fries, Wu & Ré (2017). Snorkel: Rapid Training Data Creation with Weak Supervision. VLDB. arXiv:1711.10160.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Clean records are the quiet groundwork behind a co-pilot you can just talk to. See it on a real call.
No app required. Just your phone number.