Continual learning without forgetting.
Teach a model something new and it can quietly get worse at what it already knew. That failure has a name, catastrophic forgetting, and a field has grown up to fix it.
This is an educational walk through that field, in plain language. What forgetting is, why it happens, the three families of fixes, what they make possible, and where the work is heading. The papers behind each idea are linked, so you can read the source yourself.
Learning new things over time, while keeping the old.
The field name is continual, or lifelong, learning. The failure it fights is catastrophic forgetting.
Picture a new hire who is great at one task. You train them on a second task, and somehow they get worse at the first. That is the problem at the heart of this page. When a neural network learns task B, the same internal settings that held task A get written over. The network does not file the new skill next to the old one. It writes on top.
Why does it happen? A model keeps everything it knows in one big shared set of numbers, its weights. There is no separate drawer per skill. Training nudges those numbers toward whatever it is seeing right now, so show it only new examples and the old behavior quietly degrades, with no explicit forget step needed.
The core tension to name early: stability versus plasticity. Too stable and the model cannot learn anything new. Too plastic and it forgets. The whole field lives on that trade-off. Across this page we use one running example: an assistant that has to keep up with how a company sells as products, objections, and competitors change, without unlearning what worked before, and with each account's learning kept to itself. That is an example of where the idea applies, not a claim about how any one product is built.
The same machinery that learns is what overwrites.
Forgetting is not a bug bolted on. It falls out of how a model stores knowledge and how training moves it.
No separate drawer per skill.
A model keeps all its knowledge in one set of weights. There is no slot reserved for task A, so any new training touches the same numbers the old skill depends on.
It fits whatever it sees now.
Gradient descent nudges the weights to fit the current batch. Feed it only new examples and the weights drift to fit them, and old behavior degrades as a side effect.
You cannot always re-show the past.
In many real settings you cannot keep or replay all past data, for cost, scale, or privacy reasons. So a skill has to be preserved without full access to what taught it.
Often there is no task label.
At use time the model frequently does not get a tag saying "this is task 3." Whether task identity is known, and whether it must be inferred, defines genuinely different difficulty levels.
Average accuracy can hide it.
One headline number can mask forgetting. You need to track how much each old skill drops after new learning, called backward transfer, and how new learning helps or hurts what comes next, called forward transfer.
Three families of fixes, and one question to ask first.
Protect what matters, replay the past, or grow the model. Most real systems blend them. And the right blend depends on which problem you are actually solving.
1Protect the important weights · regularization
Slow down the changes that would hurt.
Estimate which weights mattered most for the old task, then add a penalty that pulls those specific weights back toward their old values during new training. New learning flows around the protected ones instead of through them, and the old data does not need to be stored.
The reference point is Elastic Weight Consolidation (Kirkpatrick et al., 2017). The analogy the field reaches for is biological: synapses that matter for an old skill are made harder to change, so the skill survives while new ones form.
Earlier task
- Estimate importance
- Identify parameters that strongly affect the old task.
New task
- Constrain the update
- Penalize large changes to those parameters.
Fit the new examples while protecting useful earlier behavior.
2Keep the behavior, not the weights · functional regularization
Hold the old answers steady while the model moves.
Before learning the new task, record what the old model would output. Then train the new task while keeping the model's responses on old-type inputs close to that record. The weights are free to move, but the behavior is preserved.
The reference point is Learning without Forgetting (Li & Hoiem, 2016), which draws on knowledge distillation and needs only new-task data to do it.
| Example | Old model | Updated model target |
|---|---|---|
| Call A | Renewal: 0.8 | Keep a similar distribution |
| Call B | Expansion: 0.7 | Keep a similar distribution |
Illustrative outputs. Combine this distillation objective with learning the new task.
3Replay the past · rehearsal
Mix a little of the old back in.
Keep a small, carefully chosen set of old examples and mix them back into training while learning the new task, so the old skill stays fresh. A representative method also keeps a prototype for each class and classifies by nearest prototype, so it can grow to new classes over time.
The reference point is iCaRL (Rebuffi et al., 2017), which pairs a replay memory with prototype-based classification.
New stream
Replay memory
B1 · B2 · B3 · A7 · A19
4Replay as a constraint · gradient memory
Only step in directions that do not hurt.
Instead of only adding old examples to the batch, use a memory of old examples to constrain each update so it does not increase loss on past tasks. The new step is allowed only in directions that do not damage what is remembered.
The reference point is Gradient Episodic Memory (Lopez-Paz & Ranzato, 2017), which also formalizes backward and forward transfer as ways to measure retention.
- Propose an update
Compute a direction that improves the new task.
- Evaluate memory constraints
Estimate whether it would increase loss on stored earlier examples.
- Adjust if needed
Project the update into an allowed direction.
5Grow the model · architecture
Give new learning somewhere of its own to go.
Add a small amount of new capacity for the new task and freeze or shield the part that holds the old skill, so new learning has a place to land that does not overwrite the old. This trades a little extra size for protection from forgetting.
The field review Parisi et al. (2019) sorts this alongside regularization and replay as the third family, and notes that real systems often combine all three.
Existing parameters
Retain the previous representation.
New parameters
Learn the new task in added capacity.
The exact connection between the base and module depends on the architecture.
6Know which problem you are solving · scenarios
Name the setting before you pick a method.
The right method depends on what is actually changing: are you adding new tasks, new input domains, or new classes, and is the task identity given at use time or must it be inferred. Naming the scenario up front is half the work.
The reference point is the three scenarios of van de Ven & Tolias (2019), which separate task-, domain-, and class-incremental learning by whether the model is told which task it faces.
| Setting | What changes |
|---|---|
| Task-incremental | New tasks; task identity may be provided. |
| Domain-incremental | Input distribution changes for a shared task. |
| Class-incremental | New classes join the prediction space. |
The trade-off that keeps the field honest.
Every method above is a way to sit between two poles, not a way to escape them.
A system must stay flexible enough to learn yet stable enough to keep what it learned. Push all the way to stable and nothing new gets in. Push all the way to plastic and the past washes out. Protect, replay, and grow are all ways of finding a workable middle, and the right point on that line depends on the scenario you named above.
Stability
- Measure
- How well does the model retain earlier skills?
Plasticity
- Measure
- How well does it learn the new examples?
Compare performance on old and new tasks after each update.
What it makes possible.
At the field level: keep up without retraining from scratch, remember while learning, preserve skills from less stored history, add new categories over time, and adapt to a setting without disturbing the base.
The enterprise lines below are examples of where this family of techniques shows up, framed as everyday outcomes a seller would recognize. They are not claims about how any one product is built, and none of them describe what is or is not inside a given system. Keeping one account's learning separate from another's is organizational practice built around a model, stated here at the outcome level only.
Keep up without starting over.
A model can absorb new data and new tasks as the world changes, instead of being frozen at training time or fully retrained each time. In a sales setting that is the shape of keeping up with new products, objections, and competitors.
Improvement that adds up.
Old skills survive the arrival of new ones, so progress accumulates instead of trading one capability for another. What worked last quarter is still there after this quarter's lessons land.
Learn from less stored history.
Regularization and prototype methods preserve skills without keeping all the old data, which helps wherever storage or data-retention limits constrain what can be kept. Add class-incremental methods and a model can recognize new categories as they appear.
Targeted, isolated adaptation.
Growing a small amount of capacity lets a model specialize without disturbing the shared base. As an enterprise example, that is the shape of adapting one base model to one company's way of selling while keeping that adaptation isolated, stated as where the idea applies, not a product claim.
On the product itself we keep it simple: Prism is our own model, built for sales conversations. Everything else here describes the field, not our plumbing.
The direction the field is heading.
We describe the direction, not a roadmap of our own. The open problems sit alongside the progress.
Combine, do not pick one.
Real systems mix protect, replay, and grow rather than choosing a single family, tuning the blend to the scenario rather than to a favorite technique.
Stay current without full retraining.
Keeping a large model up to date by adapting small added components rather than moving all the weights, so updates stay cheap and the base is preserved.
Measure retention, not just accuracy.
Benchmarks and metrics that track real retention and transfer across a long stream, so "it kept learning" can be shown rather than asserted.
Keep the gist, not the records.
Ways to retain or regenerate the gist of past data without storing sensitive raw records, which matters wherever data cannot be kept verbatim.
How brains consolidate.
Continued cross-pollination with neuroscience on how living systems keep old memories while forming new ones, the same plasticity-stability balance from a different angle.
The papers behind this page.
- Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks (Elastic Weight Consolidation). arXiv:1612.00796.
- Li & Hoiem (2016). Learning without Forgetting. arXiv:1606.09282.
- Rebuffi et al. (2017). iCaRL: Incremental Classifier and Representation Learning. arXiv:1611.07725.
- Lopez-Paz & Ranzato (2017). Gradient Episodic Memory for Continual Learning. arXiv:1706.08840.
- Parisi et al. (2019). Continual Lifelong Learning with Neural Networks: A Review. arXiv:1802.07569.
- van de Ven & Tolias (2019). Three Scenarios for Continual Learning. arXiv:1904.07734.
Keep reading.
Sibling explainers in the Research section. Staying current has a learning side and a retrieval side.
Talk. Done.
The field is deep, but the promise is plain: you talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.