Weak supervision and data programming.
Noisy rules go in. Clean, probabilistic labels come out. Here is how a field teaches a model to clean data without labeling every row by hand.
Weak supervision is a way to build a large training set from many cheap, noisy sources of signal instead of one expensive source of hand labels. Data programming is the recipe behind it: write your rules of thumb as small programs, then let the math decide how much to trust each one. We walk it the way the field is built: what it is, why it is hard, how it works, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
“Please send pricing. The trial keeps crashing.”
- Mentions “pricing”
- Sales
- Mentions “crashing”
- Support
- Mentions “invoice”
- Abstain
Weigh rule reliability and overlap
Write down the rules you already use.
The field name is weak supervision. Data programming is the specific recipe: rules as code, then a model that weighs them.
Picture a sales database that has been lived in for a few years. The same account shows up twice, once as "Acme Corp" and once as "Acme Corporation." A contact has a name and an email but no title. A deal has sat in negotiation for eight months and nobody can say if it is real. To a person who knows the accounts, most of these are obvious. The trouble is the obvious only lives in someone's head, and there are tens of thousands of rows.
The instinct is to teach a model to clean it up. But a model learns from examples, and examples mean labels: someone has to mark "these two records are the same company" thousands of times before the model can copy the judgment. Weak supervision is the field's answer. Instead of labeling rows one at a time, you write down your rules of thumb as small programs, called labeling functions, apply them to every row, and let the math sort out how reliable each rule is. Two jobs, kept separate: writing a rule is a human job, deciding how much to trust it is a math job done from the data.
- Unlabeled examples
Records have no complete manual answer key.
- Labeling functions
Rules vote for a label or abstain.
- Label model
Combine votes under a model of reliability and dependence.
- Training set
Use probabilistic labels to train an end model.
Hand-labeling is the wall everyone hits.
A model needs many examples before it copies a judgment. Producing them by hand is the bottleneck the whole field exists to break.
Hand-labeling does not scale.
Marking "same company" or "dead deal" thousands of times is slow and costly, and it has to be redone every time the schema, the product, or the definition of a stale deal changes. That cost stops most teams before they start.
One rule is never enough.
Any single heuristic is right often and wrong sometimes. "Same website domain means same account" misses subsidiaries, and breaks on resellers and shared inboxes. You cannot ship a cleaner built on one brittle rule.
Rules vote against each other.
Different heuristics land on different answers for the same row. A naive majority vote treats a sloppy rule and a sharp one as equals, which buries the good signal under the noise.
You cannot measure what you cannot see.
You want to weigh rules by accuracy, but you do not know the true labels, so you cannot measure accuracy directly. The field's trick is to infer reliability from the pattern of agreement and disagreement, without ground truth.
Clean does not mean correct.
A combined label can be confidently wrong if every rule shares the same blind spot. Weak supervision reduces noise. It does not promise truth, and an honest account keeps those two ideas apart.
Some rows no rule touches.
Rules abstain when they do not apply, so parts of the data are left unlabeled. Getting enough coverage without sacrificing reliability is its own balancing act.
Write rules, weigh them, then train.
The standard pipeline has three boxes: weak sources, a label model, an end model. Each does one thing.
- 01Write
Rules vote or abstain.
- 02Model
Estimate reliability and dependence.
- 03Label
Keep probabilistic targets.
- 04Train
Learn an end model.
- 05Evaluate
Use independent trusted labels.
The pipeline the field survey maps: weak sources, a label model, an end model, in that order.
1Labeling functions: rules as code
Each rule votes, or abstains.
A labeling function looks at a row and either votes a label or steps aside. "Shares a website domain, so probably the same account." "Emails bounce and no activity in twelve months, so probably stale." They are allowed to be noisy, narrow, and overlapping. Coverage matters more than perfection, so you want many cheap rules rather than one perfect one. This unit of work, the rule as a small program, is the heart of data programming (Ratner et al., 2016) and the Snorkel system (Ratner et al., 2017).
| Function | Observed evidence | Vote |
|---|---|---|
| same_email | Verified emails agree | Match |
| conflicting_id | Different tax IDs | Non-match |
| address_check | Address missing | Abstain |
2A label model combines the votes
Learn whom to trust, with no answer key.
Instead of a majority vote, the system learns a weight for each rule from how its votes line up with the rest. A rule that agrees with the consensus when the signal is strong is trusted more. A rule that fires at random is trusted less. This is estimated without any true labels, from the structure of the votes alone. Data programming frames it as a generative model over the labeling process. Later work finds the weights in closed form for speed: FlyingSquid (Fu et al., 2020) uses triplet methods to solve the label model far faster than iterative fitting.
Independent evidence
- Signals
- Email agreement and a verified registry ID.
- Potential value
- Different sources support the same label.
Dependent evidence
- Signals
- Two rules both use the same email-domain test.
- Risk
- Counting the same evidence twice.
3The output is a probability
Confidence, not a verdict.
Each row comes out with a probabilistic label, not a hard yes or no. This pair is the same account at 0.95. That deal is dead at 0.6. The uncertainty is kept, not discarded, which lets the next stage learn from strong and weak evidence differently. Keeping clean shape separate from correct content is the honest part: a confident label can still be wrong if the rules shared a blind spot.
Illustrative model outputs.
4An end model learns from the labels
Generalize beyond the rules.
The probabilistic labels become the training set for a stronger, more general model. That end model can apply judgment to rows no single rule covered, which is the payoff over just running the rules directly. The rules got you a labeled set without an army of annotators. The end model is what reaches the cases the rules missed. The field survey (Zhang et al., 2022) names this final stage as the standard close of the pipeline.
- Rule-covered examples
Combine records with their probabilistic labels.
- Train the end model
Learn patterns from the example features.
- Uncovered examples
Predict on cases where the original rules abstained.
- Validate
Measure against a trusted held-out labeled set.
5Automating the rules
Edit rules instead of writing every one.
Writing every rule by hand is its own work, so newer methods generate candidate labeling functions from a small labeled set and let a person refine them. Snuba (Varma & Ré, 2018) proposes rules automatically, turning rule authoring into rule editing. More recent practice also uses large models as additional weak sources that suggest labels or rules, expanding the set of cheap signals the label model can weigh.
Initial rule
same domain → matchRevised rule
same verified email
and no conflicting ID
→ matchInspect which examples the changed rule labels.
Labeled data at the speed of writing rules.
At the field level: standing up a training set in hours instead of weeks, and re-running it when the definition changes, because the rules are code. The examples below are problem illustrations, not claims about any one product.
Record-cleaning is the kind of problem weak supervision is well suited to frame. Deduping accounts that appear under several names, matching a contact to the right company, filling gaps in sparse records, flagging a stale pipeline: each is a row whose true label nobody wrote down, and each is the sort of judgment a person can express as a handful of rules. None of the following describes how any specific system works. They show the shape of what the field makes possible.
Experts contribute directly.
The person who knows the accounts writes the rules. They no longer have to hand-annotate thousands of rows to transfer that knowledge into a training set.
Adaptable to change.
When the schema or the policy shifts, you edit a rule and regenerate labels, rather than commissioning a fresh labeling round from scratch.
Many cheap signals, combined.
Heuristics, external lists, and patterns can all act as weak sources. The label model weighs them together instead of forcing a choice of one.
Uncertainty is preserved.
Because labels come out as probabilities, downstream review can focus on the rows the rules were least sure about, where a human check pays off most.
This page explains the field. It makes no claim about how any particular product cleans data or builds its training sets.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Denoising in closed form.
Triplet methods replaced slow iterative fitting, making the weighting step quick on large rule sets, as in FlyingSquid (Fu et al., 2020).
Rules that propose themselves.
Automatic generation and refinement of labeling functions, building on Snuba, plus large models acting as additional weak sources.
Shared benchmarks and a map.
The field now has common benchmarks and a survey that charts the standard pipeline, a sign it has settled into named, comparable methods.
Probabilistic, never certain.
Soft labels are not ground truth. Shared blind spots produce confident errors, and coverage gaps leave rows unlabeled. Weak supervision lowers the labeling cost. It does not remove the need to check results against reality.
The papers behind this page.
- Ratner et al. (2016). Data Programming: Creating Large Training Sets, Quickly. arXiv:1605.07723.
- Ratner et al. (2017). Snorkel: Rapid Training Data Creation with Weak Supervision. arXiv:1711.10160.
- Varma & Ré (2018). Snuba: Automating Weak Supervision to Label Training Data. PVLDB 12(3).
- Fu et al. (2020). Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods (FlyingSquid). arXiv:2002.11955.
- Zhang et al. (2022). A Survey on Programmatic Weak Supervision. arXiv:2202.05433.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
You talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.