Matching records without exposing PII.
Scattered, conflicting records go in. One clean answer comes out, without putting names, emails, and phone numbers on the table for everyone to read.
Record linkage is the field that decides which records describe the same real-world entity when no shared ID exists and the fields disagree. Privacy-preserving record linkage adds a constraint: do the matching without revealing the identifying values you match on. We walk it the way the field is built: what it is, why it is hard, how it works, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Deciding which records are the same entity.
The field name is record linkage. It also goes by entity resolution, data matching, and deduplication.
Every team that keeps customer records ends up with the same mess. The same account is in the system twice, once as "Acme Inc." and once as "ACME Incorporated." A contact is half filled in: a name and a company, but no email. A pipeline goes stale because two reps were quietly working the same logo under different spellings. Somewhere there is a clean answer, but the records do not line up on any single exact field, so the duplicates and gaps survive. That everyday pain is the same problem researchers formalize.
Record linkage decides which records, across one or more datasets, refer to the same real-world entity even when no shared ID exists and the fields disagree. Deduplication is the within-one-dataset case, the account that exists twice. Linkage is the across-datasets case, matching a marketing list to the records you already hold. Either way the unit of work is the record pair: the field reasons about pairs of records and reaches a decision per pair.
Privacy-preserving record linkage (PPRL) adds a second question on top of the first. Not just which records match, but who is allowed to see what during the matching. It is record linkage done so the parties do not expose the identifying values they compare on. That second question is what the rest of this page builds toward. These CRM examples are just relatable versions of the field's problem, not a description of how any particular product works.
- Prepare locally
Normalize permitted identifying fields.
- Compare under a protocol
Use protected representations or secure computation.
- Return the allowed result
For example, matched record IDs rather than raw personal fields.
Real records rarely agree on a single exact field.
Identity hides in noisy combinations, the comparisons do not scale on their own, and hiding the values fights the very signal matching depends on.
Equality is not enough.
Typos, abbreviations, nicknames, reordered name parts, changed addresses, and formatting differences mean the same entity rarely matches on a literal equality test. Matching has to tolerate near-agreement, not just equality.
Comparing everything is quadratic.
Comparing every record to every other does not scale. A list of 100k against another 100k is ten billion pairs. That is why blocking and indexing exist, and why scale is a core research problem rather than an afterthought.
Identity hides in combinations.
No single field is decisive. A name agrees but is common. An email agrees and is nearly conclusive. Fields that identify only in combination are called quasi-identifiers, which is why "just strip the name" does not protect a record on its own.
Hiding the value blurs the signal.
Encoding or encrypting identifiers to protect them tends to blur the very signal matching depends on. Stronger hiding can mean more false matches or more misses. The job is to make that trade-off explicit and tunable, not to pretend it is free.
Encodings can be attacked.
Privacy-preserving encodings are not automatically safe. Frequency analysis and pattern attacks can partially reverse naive Bloom filter encodings, so the literature pairs each encoding scheme with the cryptanalysis that hardens it. We name that honestly.
Borderline pairs are real.
Many pairs are neither a clear match nor a clear miss. A principled method surfaces that middle band for review instead of silently merging two accounts or dropping a real duplicate.
Weigh the agreement, shrink the haystack, then hide the values.
A probabilistic core scores each pair. Blocking makes it tractable. Encodings and cryptographic protocols let the matching happen without exposing the underlying details.
- 01Agree
Define permitted inputs and outputs.
- 02Prepare
Normalize fields locally.
- 03Protect
Use the chosen comparison protocol.
- 04Compare
Evaluate candidate pairs.
- 05Release
Reveal only the agreed result.
A common shape: narrow the pairs, compare the fields, weigh the score, protect the values.
1The probabilistic core
Weigh each field, then make a three-way call.
The foundational model is Fellegi and Sunter (1969). Compare two records on a set of fields. Turn each field's agreement or disagreement into a weight: how likely that agreement is among true matches versus among non-matches. Sum the weights into one score, then apply two thresholds.
That yields three outcomes, not two: link, non-link, and possible-link, where the pair is sent to a human. The middle band is the point. A weighted score with an explicit review zone surfaces borderline pairs instead of forcing a brittle exact-match rule to guess. Modern matching still builds on this backbone.
| Evidence | Candidate outcome |
|---|---|
| Several reliable identifiers agree | Link |
| Name agrees, other fields missing | Review |
| Reliable identifiers conflict | Non-link |
2Fuzzy comparison and blocking
Measure near-agreement, and only where it is worth it.
Field comparison uses string similarity, edit distance, Jaro-Winkler, and token or phonetic methods, so "Acme Inc." and "ACME Incorporated" register as near-agreement rather than a flat mismatch. That is the fuzzy step exact joins lack.
Then blocking and indexing keep the work tractable. Instead of comparing every record to every other, compare only records that share a blocking key: a postcode, a sorted-name fragment, a phonetic code. That is what turns a quadratic problem back into something you can run at scale.
All pairs
Candidate blocks
3Encoding for privacy
Compare the bits, not the names.
The reference introduction of privacy-preserving encoding is Schnell, Bachteler, and Reiher (2009). Split each identifier into character n-grams, hash them into a bit vector (a Bloom filter), and compare two encoded records by the overlap of their set bits.
The point is that approximate similarity survives the encoding, so fuzzy matching still works, while the raw values are not transmitted in the clear. The honest caveat stays attached: naive encodings can be partially reversed by frequency and pattern attacks, which is why the same line of work also produced the hardening that followed.
- Normalize and split
For example, turn a name into character q-grams.
- Encode
Set positions in a Bloom-filter bit vector.
- Compare
Measure overlap between protected representations.
Frequency patterns and similarities can permit attacks on poorly protected encodings.
4The cryptographic route
Both sides learn what they share, and nothing else.
Private set intersection (PSI) takes a different path. Two parties each hold a set, and the protocol lets them compute, or just count, the intersection while neither learns anything about the elements not in common. The foundational protocol, built on homomorphic encryption, is Freedman, Nissim, and Pinkas (2004).
Early protocols were costly. The oblivious-transfer-extension line in Pinkas, Schneider, and Zohner (2014) reports large speedups over earlier work, pushing PSI from theory toward practical scale. On architecture, the field describes two design points neutrally: setups that use a trusted third party that only ever sees encoded values, and two-party protocols with no third party at all.
Party A
Party B
The protocol reveals the agreed output to the designated party or parties.
Clean, joined records without a central data dump.
At the field level: finding an overlap without handing over everything in the clear, deduping and gap-filling on sensitive data responsibly, and cross-organization joins that raw identifiers could not allow.
The everyday CRM pains map straight onto the field's payoff. The same machinery that collapses a double-listed account, reconciles a half-filled contact, or untangles a pipeline counted twice under two spellings is the matching the literature formalizes. Stated as a field capability, not a claim about how any product does it.
Deduping without exposing the rows.
Two systems can agree that two records are the same entity, or find their overlap, while running on encoded identifiers rather than raw names and emails. See Record normalization.
Gap-filling on personal data, responsibly.
A blank email on one record can be reconciled against a second record that holds it, with the matching done on encoded values to reduce what is exposed to do the join.
Measured matching, not guesswork.
The weighting gives a principled score and an explicit review band, so borderline pairs are surfaced rather than silently merged or dropped.
Cross-organization collaboration.
Health, finance, and research routinely need to link datasets across institutions that cannot legally pool raw identifiers. PPRL is what makes that join possible at all.
Everything here describes the field. We make no claim about how Prism, or any of our own systems, performs matching, deduping, or encoding.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
More parties, larger sets.
Multi-party PPRL, counting Bloom filters, and faster PSI protocols keep pushing the size and party-count limits of what can be matched at once.
Representation learning for matching.
Newer work explores representation-learning and embedding encodings, aiming for better accuracy on messy fields while keeping a privacy posture.
Encodings paired with their attacks.
An active back-and-forth between encoding schemes and cryptanalysis means modern practice pairs every encoding with its known attacks and countermeasures.
A maturing common taxonomy.
The survey work of Vatsalan, Christen, and Verykios (2013) gives practitioners a shared map to compare methods on accuracy, scalability, and privacy on equal terms.
No method is free.
Every privacy gain costs some matching signal, encodings can be attacked, and a match score is a probability, not a certainty. Confidence with candor is the honest close.
The papers behind this page.
- Fellegi, I. P. & Sunter, A. B. (1969). A Theory for Record Linkage. Journal of the American Statistical Association.
- Schnell, R., Bachteler, T. & Reiher, J. (2009). Privacy-preserving record linkage using Bloom filters. BMC Medical Informatics and Decision Making.
- Freedman, M. J., Nissim, K. & Pinkas, B. (2004). Efficient Private Matching and Set Intersection. EUROCRYPT 2004.
- Pinkas, B., Schneider, T. & Zohner, M. (2014). Faster Private Set Intersection Based on OT Extension. USENIX Security 2014.
- Vatsalan, D., Christen, P. & Verykios, V. S. (2013). A taxonomy of privacy-preserving record linkage techniques. Information Systems.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
You talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.