Filling in missing data.
The same account twice. A contact with half its fields blank. A deal still marked open after it closed. Here is the field that does something rigorous about that.
Filling in missing data covers two linked jobs: matching, which decides which records describe the same real-world thing so duplicates can be merged, and imputation, which fills the fields still blank with principled estimates. We walk it the way the field is built: what it is, why the gaps are not random, how serious methods work, what clean records unlock, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
| Account | People | Value |
|---|---|---|
| Acme | 48 | Observed |
| Northwind | 64 | Observed |
| Oakline | ? | Missing |
Observed values stay the same in every version.
Repairing records that are spread out or incomplete.
A record is a row that is supposed to describe one thing. Real datasets break that in two ways: one thing spread across several rows, and one row missing several fields.
Open any database that real people type into and you find three kinds of holes. The same account shows up twice, once as "Acme Corp" and once as "Acme Corporation," each holding half the story. A contact has a name and a company but no title, no region, no last-touch date. A deal sits in the pipeline marked "open" that the team knows closed two quarters ago. The records are not wrong on purpose. They are just incomplete, and incomplete records quietly break the things built on top of them: forecasts, routing, reporting, follow-up.
The field has two halves that work together. Matching, also called record linkage, figures out which rows are really the same thing, so the two "Acme" rows can be recognized as one account and merged. Imputation fills the gaps that remain, so a blank field gets a careful estimate rather than a guess. The point the field is careful about: a filled-in value is an estimate, not a recovered fact. A good method does not just paste something in. It weighs every other record like this one, works out the most defensible value, and the better methods can also say how sure they are.
- Link records
Confirm which rows describe the same entity.
- Merge observations
Fill a gap from another verified source when possible.
- Impute remaining gaps
Estimate from the available data and assumptions.
- Mark the estimate
Retain uncertainty and provenance.
The gaps are not random, and the duplicates hide them.
Most of the difficulty is in two questions: why is this value missing, and which other rows are really the same thing.
A blank carries its own pattern.
The fields that go missing tend to be the ones people skip when they are busy, so the gaps are not a random slice. The reason a value is missing changes how you should fill it. Rubin (1976) named the three cases the field still uses. Pasting in an average ignores all three and quietly biases the result.
Duplicates hide the gaps.
Two half-filled rows for one account each look incomplete, but together they may hold the full picture. You cannot fill a field correctly until you know which other rows are the same entity. That is why matching runs before imputation.
Matching is rarely clean.
"Acme Corp" versus "Acme Corporation," a typo in an email, a phone number that moved. The method has to weigh partial agreement across many fields and decide link, no-link, or send it to a human.
A single fill hides its own uncertainty.
Filling each blank with one best guess and then analyzing the table as if it were complete makes everything downstream look more certain than it is. A forecast built that way reports false precision.
Filling can invent structure.
A method that learns the patterns in the data can also reproduce and amplify them, including stale or skewed ones. A filled value should never be presented as a recovered fact, especially when the reason for the gap was misread.
Throwing out incomplete rows is its own bias.
Many tools quietly discard rows with any blank. When the incomplete rows are not a random slice, dropping them skews the result just as much as a careless fill would.
Not every blank is the same blank.
The whole "gaps are not random" idea rests on three cases the field names, each with an everyday example. The source is Rubin (1976).
Missing completely at random.
The blank has no pattern at all, like a field a system dropped here and there for no reason tied to the data. The friendliest case, and the rarest in records people type by hand.
Missing at random.
The blank can be predicted from other fields you do have, like reps in one region routinely skipping a field. Knowing the region tells you a lot about why the value is gone, which is exactly what a principled method uses.
Missing not at random.
The blank depends on the value itself, like the largest deals being the ones left un-updated. The hardest case, because the reason for the gap is tangled up with the answer you are trying to estimate.
Match and merge, then fill, then carry the uncertainty forward.
Decide which rows are the same entity, fill the fields still blank from everything similar, and keep a record of how sure each estimate is.
- 01Compare
Find plausible duplicate pairs.
- 02Decide
Link, review, or separate.
- 03Merge
Preserve observed evidence.
- 04Estimate
Impute remaining gaps.
- 05Disclose
Keep uncertainty visible.
The two halves in order: matching resolves the entities, imputation fills what remains, and the spread is carried, not discarded.
1Matching, classically
Treat each pair of records as a question.
The foundational frame is Fellegi and Sunter (1969). For each candidate pair of records it asks: given how much the fields agree, is this a match. It compares fields, weighs each agreement and disagreement by how informative it is, and sorts pairs into three buckets: link, non-link, and a middle band sent for review. Much of modern matching still rests on this idea.
The middle band matters. Instead of forcing every borderline pair to a yes or a no, the method sets it aside as needs-a-look. That is where a person belongs, exactly on the pairs the data cannot settle on its own.
| Evidence | Decision |
|---|---|
| Different verified identifiers | Keep separate |
| Similar names, incomplete identifiers | Review |
| Same verified identifier, compatible details | Link |
2Matching, learned
Let a model judge whether two records are the same.
Newer work casts matching as a learned task: represent each record, then train a model to predict whether two records describe the same entity. The design-space study Mudgal et al. (2018) maps how the representation and comparison choices trade off, and shows learned matchers help most on noisy text fields where simple rules miss, like free-typed company names and addresses.
This is the modern counterpart to the classic frame, not a replacement for the idea behind it. The question is still link or no-link. What changed is that the comparison can be learned from examples rather than spelled out by hand.
Record A
Acme Corp
phone: 555-0118Record B
Acme Corporation
phone: 555 0118Normalize and compare fields, then evaluate conflicting evidence.
3Imputation, one column at a time
Fill each blank several ways, not once.
A widely used family fills each incomplete column by predicting it from the others, looping over columns until the estimates settle. The MICE approach van Buuren and Groothuis-Oudshoorn (2011) does this and, importantly, fills each blank several times to produce several complete versions of the table. Analyzing all of them and pooling the answers carries the uncertainty through instead of hiding it. This is the core idea of multiple imputation.
The contrast is worth holding onto. A single pasted value gives one falsely sharp answer. Several principled fills give several answers whose spread is the honest signal: where they agree, you can lean on the number, and where they scatter, you know to be careful.
Pool the analysis estimates and uncertainty across imputations.
4Imputation with machine learning
When fields are tangled and mixed-type.
When the relationships between fields are tangled and the data mixes numbers and categories, tree-based methods fit well. missForest Stekhoven and Bühlmann (2012) imputes by repeatedly training a random forest to predict the missing entries from the observed ones, handles mixed data without much tuning, and reports an estimate of its own error from the model's out-of-bag samples. That self-reported error is a strong baseline trait: the method tells you roughly how far off it expects to be.
| Field | Available information |
|---|---|
| Region | West |
| Company size | 200 |
| Industry | Manufacturing |
| Segment | Missing |
An estimate from patterns in observed records, not a recovered fact.
5Imputation with generative models
Learn the whole shape of the data.
A more recent line treats filling blanks as a generation problem. GAIN Yoon, Jordon, and van der Schaar (2018) adapts the generative-adversarial setup: one network fills the missing entries, another tries to tell filled entries from real ones, and the contest pushes the filled values toward the true data distribution. It is a clear marker of how the field moved from per-column models toward learning the joint shape of a dataset at once.
The trade is real. Joint models can respect relationships across many fields that a column-by-column method handles one at a time, but they ask for more data and more care to train. The right tool depends on how tangled the data is and how much of it there is.
Conditional approach
Predict segment from
region, size, industryJoint approach
Model the distribution of
region, size, industry, segmentGenerate plausible values for those that are missing.
6Putting the two halves together
The running example, resolved.
In plain terms: match and merge first so each entity is one record, then impute the fields that are still blank, then carry the uncertainty forward. The CRM example from the top resolves here. The two "Acme" rows are recognized as one account and merged. The blank title and region are estimated from similar contacts and marked as estimates. The deal still marked open is flagged for a human rather than silently trusted, because a stale field is the kind of gap a fill should not paper over.
| Field | Value | Status |
|---|---|---|
| Company | Acme Corporation | Verified source |
| Region | West | Observed |
| Headcount | About 192 | Estimated |
| Contact status | Needs review | Possibly stale |
Clean, complete records are raw material for everything built on top.
At the field level: one record per thing, usable fields where there were blanks, numbers that carry their own confidence, and analysis that does not silently drop rows.
Forecasts, routing, reporting, and follow-up all sit on top of the records underneath them. When the same account is two half-filled rows, every layer above inherits the mess. Resolve the entity and fill the gaps in a principled way, and the layers above get something they can actually trust. The harder cases in the section above are exactly the ones that decide whether a messy export becomes a record a team can work from.
One record per thing.
Merged accounts and matched contacts mean reports, routing, and follow-up act on a single clean entity instead of fighting duplicates.
Usable fields where there were blanks.
Estimated titles, regions, and segments make filtering and reporting possible on records that were dead weight before.
Honest numbers.
Multiple imputation lets a forecast or a count come with a sense of how solid it is, instead of a single figure that only looks precise.
Rows that are not silently dropped.
Principled imputation keeps incomplete rows in the analysis, which matters most when the incomplete ones are not a random slice.
Everything on this page describes the field of matching and imputation, not the plumbing of any one product. A filled value is an estimate, and a careful method keeps it labelled as one.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Learn the whole dataset at once.
The trend is toward models that learn the shape of a dataset as a whole rather than one column at a time, so a fill respects relationships across many fields.
Matching and filling, closer together.
Treating dedup and fill as one pipeline rather than two disconnected steps, since each one helps the other do its job.
Report a value and how sure it is.
More methods return not just a value but a measure of confidence, so a person can be pulled in exactly where confidence is low.
A filled blank is still an estimate.
The field's own literature is clear that imputation can bias results when the missingness reason is misread, and that a filled value should never be presented as a recovered fact.
Holding up on free-typed text.
Better matching on the messy, free-typed company names, addresses, and emails that simple rules miss, across larger and noisier datasets.
The papers behind this page.
- Rubin (1976). Inference and Missing Data. Biometrika 63(3), 581-592. DOI 10.1093/biomet/63.3.581.
- Fellegi and Sunter (1969). A Theory for Record Linkage. Journal of the American Statistical Association 64(328), 1183-1210.
- van Buuren and Groothuis-Oudshoorn (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 45(3), 1-67. DOI 10.18637/jss.v045.i03.
- Stekhoven and Bühlmann (2012). MissForest: non-parametric missing value imputation for mixed-type data. Bioinformatics 28(1), 112-118. Preprint arXiv:1105.0828.
- Yoon, Jordon, and van der Schaar (2018). GAIN: Missing Data Imputation using Generative Adversarial Nets. ICML, PMLR 80, 5689-5698. Preprint arXiv:1806.02920.
- Mudgal et al. (2018). Deep Learning for Entity Matching: A Design Space Exploration. SIGMOD. Open PDF deepmatcher-sigmod18.pdf.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Clean, complete records start with a clean, complete note. You talk, and the record is waiting. See it on a real call.
No app required. Just your phone number.