Fuzzy matching and blocking at scale.
Many messy rows go in. One clean, resolved record comes out. Here is the field that makes that happen, in plain language.
Two records can describe the same real thing and still look different on the page. Fuzzy matching, also called record linkage or entity resolution, is the family of methods that decides whether they point at the same entity even when the text does not agree. We walk it the way the field is built: what it is, why it is hard, how a modern system does it, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
| Field | Record A | Record B |
|---|---|---|
| Name | Jon Smith | John Smyth |
| Company | Acme | Acme |
| Office phone | 555-0142 | 555-0142 |
Deciding when two records are the same real thing.
The field name is record linkage, or entity resolution. Fuzzy matching is the everyday name for the same job.
"Acme Corp," "ACME Corporation," and "Acme Corp." are one account to a person and three rows to a database. A contact named "Jon Smith" and another named "John Smyth" at the same company might be one human with a typo between them. Exact-match logic, the kind that only links two rows when every character agrees, treats all of these as strangers. So a pipeline fills with the same account twice, a half-filled contact sitting next to a complete one, and a deal that looks stale only because the fresh activity landed on the duplicate.
Fuzzy matching is the family of methods that decides whether two records point at the same real-world thing even when the text does not match exactly. It rests on two questions. How close are two values? And, across all the fields, is that enough evidence to call it a match? The line that matters: this is matching by likeness, not by exact equality.
- Candidate pair
Two records might describe the same organization.
- Compare fields
Measure spelling, identifiers, addresses, and other evidence.
- Decide
Link, keep separate, or send for review.
The same thing is written many ways, and the comparisons explode.
Deciding "same or not" for one pair is cheap. Doing it well, for every pair in a large database, is where the difficulty lives.
One entity, many spellings.
Abbreviations, typos, missing fields, reordered words, nicknames, and stale values all break exact matching. "Acme Corp" and "ACME Corporation" are one account written two ways.
Agreement has to be weighed, not counted.
Agreeing on a rare company name is strong proof. Agreeing on "John" is weak. A match decision has to weight each field by how much it actually discriminates.
The bar is a real trade.
Set it too low and you merge two different companies, a false match. Set it too high and you leave the same account split in two, a missed match. The field treats this as a tunable decision, not a fixed rule.
Comparing every pair is infeasible.
The number of pairs grows with the square of the row count. A file of a million contacts holds on the order of five hundred billion possible pairs, and almost none of them match. Brute force is accurate and unusable.
Distance math gets expensive too.
When records are turned into vectors so "similar" becomes "near," naive distance over many dimensions gets slow, the curse-of-dimensionality problem the nearest-neighbor literature names.
Linking is not a string of yes/no calls.
If A matches B and B matches C, the system has implied A matches C. Resolving a whole group of records into one entity is a clustering problem, handled deliberately rather than by accident.
A modern system resolves records in stages.
Measure closeness, score the evidence, filter the comparisons first, then stitch the survivors into single entities.
- 01Block
Choose plausible candidate pairs.
- 02Compare
Measure field similarities.
- 03Score
Combine the evidence.
- 04Decide
Link, review, or non-link.
- 05Cluster
Validate groups before merging.
The resolution pipeline the blocking survey frames: blocking runs first, then the expensive scoring runs only on a short list.
1String distance
Measuring closeness.
Edit distance, also called Levenshtein distance, counts the insertions, deletions, and substitutions that turn one string into another, so "Smith" to "Smyth" is one edit. These are the rulers the rest of the system reads.
For short fields like names, the Jaro and Jaro-Winkler comparators score partial agreement and give extra weight to a shared prefix, on the observation that strings agreeing on their first few characters are more likely to be variants of one value. The comparator was introduced by Winkler (1990) inside the record-linkage model.
Value A
Value B
Substitute I with Y. The other four letters stay the same.
2Scoring a pair across fields
Weigh the evidence, then decide.
The Fellegi-Sunter theory of record linkage (1969) frames the decision as weighing agreement and disagreement across fields, producing three outcomes: link, non-link, and a possible-link zone sent for review. This is the math under most classic deduplication.
The key idea is that each field contributes evidence in proportion to how much it discriminates, so a rare company name counts for more than a common first name. The field weights and scores shown here are illustrative sample data, not a rating of any kind.
| Field | Evidence | Relative value |
|---|---|---|
| Company | Acme Corp ↔ ACME Corporation | Supports match |
| First name | Jon ↔ John | Weak alone |
| Same verified address | Strong here |
Keep a review band for pairs that are not settled by the evidence.
3Blocking
Only compare pairs that could plausibly match.
Scoring one pair is cheap. Scoring all pairs is quadratic and infeasible at real size. So a fast filter runs first. Blocking groups records by a cheap key, say the same postal code, the first few characters of a company name, or a phonetic code for a surname, and only compares records inside the same group.
It trades a small chance of missing a match for a large cut in comparisons, turning the quadratic problem into something tractable. The survey by Papadakis and colleagues (2020) organizes the modern variants into blocking, filtering, and hybrid families.
Illustrative, disjoint blocks of equal size.
4Locality-sensitive hashing
Blocking by similarity.
When records are represented as sets or vectors, locality-sensitive hashing, or LSH, hashes similar items into the same bucket on purpose, so candidate pairs fall out of the hash table without an all-pairs scan.
Indyk and Motwani (1998) introduced LSH as a way to find near-neighbors in high dimensions in sub-linear time, which is the bridge from classic blocking keys to similarity-based candidate generation. The meaning-vector and approximate-nearest-neighbor machinery has a page of its own: Embeddings and vector search.
| Hash table | Bucket | Candidates |
|---|---|---|
| Table 1 | Bucket 7 | Record A · Record B |
| Table 2 | Bucket 3 | Record A · Record C |
Compare A with B and C using the full matching method.
5From pairs to entities
Clustering the matches into one record.
Pairwise match scores are stitched into resolved entities by grouping connected matches, so "three Acme rows" collapse into one account and conflicting field values get reconciled.
This is where transitivity is handled deliberately: if A and B linked, and B and C linked, the three belong to one group. The half-filled contact and the complete one resolve into a single richer record, so the email from one row and the phone from the other end up together.
Record A ↔ Record B
Record B ↔ Record CCheck for contradictions across the group before merging its fields.
6Learned matching
Learning the match function from examples.
Newer systems learn the match function from labeled examples instead of hand-tuned rules. Mudgal and colleagues (2018) mapped a design space for deep-learning entity matching and reported that it helps most on textual and "dirty" data, where surface text varies a lot, while structured, clean data is already handled well by classic methods.
That is useful framing for when learned matching earns its cost. Putting it together: blocking produces candidate pairs, a comparator scores each pair across fields, a model or threshold labels match, non-match, or review, and a clustering step resolves the survivors into single entities. Same fields, same rules on both sides.
| Training pair | Label |
|---|---|
| Acme Corp / ACME Corporation, same verified ID | Match |
| Acme / Acme Holdings, different IDs | Non-match |
| Bob Smith / Robert Smith, incomplete identifiers | Needs evidence |
Clean, linked records are raw material for everything built on top.
At the field level: deduplicated accounts and contacts, gap-filling by merge, an accurate pipeline, reliable matching across systems, and suppression done right.
This family of techniques is how the industry turns many messy rows into one trustworthy record. Here is what that looks like as an everyday outcome for a seller, stated as outcomes, not as a claim about how any one system is built.
The same account stops showing up twice.
One row per real company or person, so a rep is not working two copies of one deal.
A half-filled contact and a complete one read as one person.
The email from one row and the phone from the other end up together, so the right details are in one place.
The pipeline reflects real activity.
A deal stops looking stale when activity logged against a duplicate is linked back to the real opportunity, which is what makes reporting trustworthy.
Records line up across systems.
A CRM, a marketing platform, and a billing system can resolve to the same entity even when each names it differently. Resolving a person across rows is also what lets a "do not contact" flag cover every copy.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Group by meaning, not a literal key.
Representing records as meaning-vectors and blocking with approximate nearest-neighbor search, so candidate pairs are grouped by semantic closeness. See Embeddings and vector search for the vector machinery.
Match across sources that do not line up.
Matching across systems whose fields differ, without hand-built keys for every pair of sources.
Keep the possible-link zone explicit.
Holding the possible-link zone from Fellegi-Sunter as an explicit queue, so uncertain merges get a person's eyes instead of a silent guess.
Match without exposing raw identifiers.
Matching records held by different parties without revealing the underlying identifiers, an active line in the record-linkage literature.
Show why two records were linked.
Surfacing which fields agreed and how strongly, so a merge can be trusted and audited rather than taken on faith.
The papers behind this page.
- Fellegi and Sunter (1969). A Theory for Record Linkage. Journal of the American Statistical Association, 64, 1183-1210.
- Winkler (1990). String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. ASA Survey Research Methods (ERIC ED325505).
- Indyk and Motwani (1998). Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality. STOC, pp. 604-613.
- Papadakis, Skoutas, Thanos, and Palpanas (2020). Blocking and Filtering Techniques for Entity Resolution: A Survey. ACM Computing Surveys. arXiv:1905.06167 (preprint titled "A Survey of Blocking and Filtering Techniques for Entity Resolution").
- Mudgal et al. (2018). Deep Learning for Entity Matching: A Design Space Exploration. SIGMOD, pp. 19-34.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
That is the whole field in one promise: many messy rows become one record you can trust. See a real call for yourself.
No app required. Just your phone number.