Long-context memory and compression.
A language model reads everything you give it through a window. That window is its working memory. Here is the field that grows it, focuses it, and shrinks what it has to hold.
A model can use whatever is inside its context window and forgets what falls outside it, the way a person forgets the start of a long meeting by the end. We walk the field the way it is built: what the window is, why it was hard to grow, how attention is made cheaper and more selective, how a trained window gets stretched, how context gets compressed so memory survives, and what is still unsolved. The papers behind each idea are linked, so you can read the source yourself.
“Legal review comes before procurement.”
“Dana owns the renewal.”
New question: “Who should I ask about next steps?”
The context window is the model's working memory.
It is the span of text the model can read all at once while it answers. Anything inside it can be used. Anything outside it is not.
A plain analogy: a long meeting. What is in the room can be used. What was said before you walked in is not, unless someone wrote it down. The window works the same way. There are really two ingredients to "holding more": how much text fits in the window, and how well the model actually uses what is in there. They are not the same thing, and that gap runs through the rest of this page.
As the model reads, it stores a small record for every word, layer by layer, so it does not recompute the past at every step. Those running notes are called the key/value cache, and they are most of the memory cost. Carry one running example through the page: a long working session that spans several calls and documents, where the start still needs to matter at the end.
- Input context
Conversation turns, instructions, and retrieved documents.
- Model processing
Represent tokens and their positions.
- KV cache during generation
Keep attention keys and values needed by later decoding steps.
Length is expensive in compute and in memory at once.
A bigger window costs more on two fronts, and a bigger window is not the same as better memory.
Attention scales badly.
The core operation compares every word to every other word. Double the text and the work roughly quadruples. That quadratic growth is the wall that kept windows small.
The notes pile up.
The key/value cache grows with the sequence length and the number of layers. Longer conversations therefore need more memory, leaving less room for other sessions. The exact cost depends on the model, the cache precision, and how many requests run together.
Position runs out.
A model is trained to understand positions up to a certain length. Feed it positions past that and behavior can break, because it is reading coordinates it never saw in training.
A long window is not better memory.
Even with a long window, models tend to use the beginning and the end well and lose track of the middle, and accuracy can fall as the input grows. More room to read is not automatically more understanding.
Throwing it all in is wasteful.
Much of a long input is filler. Paying full attention to every word of it spends compute and memory on text that does not change the answer.
The field pushes on length from a few directions at once.
Make the core operation cheaper. Pay attention selectively. Stretch a trained window. Compress what you keep. And design around the known limit.
1Make the core operation cheaper to run
Same answer, less time and memory.
The same attention math can be computed in a way that reads and writes far less to slow memory, by working in small tiles that stay in fast on-chip memory. This is an exact result, not an approximation: the answer does not change, but longer text becomes affordable. The reference point is FlashAttention (Dao et al., 2022), which is built around being aware of how memory is actually read and written on the hardware.
Frequent transfers
- Read from slow memory
Move intermediate data repeatedly.
- Compute
Write large intermediates back.
Tiled computation
- Load a tile
Keep a smaller working set close to computation.
- Accumulate
Reuse it and combine partial results.
2Pay attention selectively
Look closely nearby, plus a few anchors.
Instead of comparing every word to every other word, let each word look closely at a nearby window plus a few globally important anchors. Cost then grows roughly in line with the length of the text rather than its square, so documents of many thousands of words become tractable. The reference point is Longformer (Beltagy et al., 2020), which pairs a local window with a handful of global positions.
All-to-all
Local + anchor
3Stretch a trained window
Rescale the positions, do not break them.
Rather than feeding the model positions it never saw, gently rescale the position information so a longer input maps back into the range the model was trained on. A model trained for a short window can then take in a much longer one with light fine-tuning, instead of breaking on coordinates it never learned. The reference point is Position Interpolation (Chen et al., 2023).
Training range
Longer sequence
Rescale or otherwise modify how the longer positions are represented.
4Compress what you keep
Keep the gist, not every word.
Two complementary ideas. The first is to remember by summary, not transcript: keep a compact, learned summary of older text instead of every original word, so the gist of the distant past survives in a much smaller footprint. The reference point is Compressive Transformers (Rae et al., 2019).
The second is to compress the instructions: train the model to fold a long, reused prompt into a handful of learned summary tokens that stand in for it, then cache and reuse those. The model behaves as if the full text were present while reading far fewer tokens. The reference point is gist tokens (Mu et al., 2023).
Several earlier calls and long email threads.
Pilot: 40 seats. Jon owns security review. Revised quote due Friday.
Keep links to original evidence for details that the summary omits.
5Know the limit, design around it
Place what matters where the model reads reliably.
Because the middle of a long input is the weak spot, where you place the important material in the window matters. This is why long context and retrieval are partners, not rivals: pull the few passages that matter, then place them where the model reads reliably. The reference point is Lost in the Middle (Liu et al., 2023). The retrieve-then-read partnership gets its own treatment in Retrieval-augmented generation.
| Placement | Same test fact |
|---|---|
| Beginning | Put the relevant fact near the start. |
| Middle | Move it into the middle. |
| End | Move it near the end. |
Keep the question and evidence constant while changing placement.
Longer working memory that stays affordable and survives.
At the field level: a model can read more, read it for less, reach past its trained limit, and keep the important parts of the past in a small footprint.
This family of techniques is how the industry lets an assistant hold a long thread and carry the important parts forward. Here is what that looks like as an everyday outcome for a seller, stated as outcomes and not as mechanism.
It remembers across a long call, and across calls.
A long thread can be held and carried forward, so you do not repeat yourself and the start of a session still matters at the end. See Deep context.
The account keeps its thread on a handoff.
When a person moves on, the context that mattered does not have to be lost with them.
The relevant history surfaces when it matters.
You get the few things that count at the right moment, not a wall of everything, which is the retrieve-then-read partnership in practice.
Affordable length, not just possible length.
Cheaper, more selective attention and compression are what turn "too expensive to read" into "fine to read," so more context is practical.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Long window plus retrieval.
Combine a long window with retrieval, so the system pulls the few passages that matter and places them where the model reads them reliably.
Compress, prune, or page the notes.
Work on the running notes, the key/value cache, so very long sessions stay within memory instead of filling it.
Recall across the whole window.
Tests that measure real recall across all of a long input, not just whether a fact can sit at the edges, so "long" means "usable."
Curated, not just a longer buffer.
Mechanisms that decide what to keep, summarize, and recall across sessions, so memory is curated rather than only a bigger window.
Built for length by design.
Continued work on attention variants and sequence models that aim to make long inputs cheap from the start.
The papers behind this page.
- Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135.
- Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150.
- Chen et al. (2023). Extending Context Window of Large Language Models via Position Interpolation. arXiv:2306.15595.
- Rae et al. (2019). Compressive Transformers for Long-Range Sequence Modelling. arXiv:1911.05507.
- Mu et al. (2023). Learning to Compress Prompts with Gist Tokens. arXiv:2304.08467.
- Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
A co-pilot that holds the thread of a long call, and carries it across calls, is what this field makes possible. To feel it, hear it on a real call.
No app required. Just your phone number.