Low-latency inference: speculative decoding and serving.
A language model writes one word at a time. Two things decide how that feels: how fast the first word arrives, and how fast each word comes after. This is the field that works on both.
Low-latency inference is the set of techniques that make a large language model reply quickly and keep the words flowing, for many conversations at once. We walk it the way the field is built: what it is, why it is hard, how a modern serving stack does it, what it makes possible, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Start sooner. Keep the rest flowing.
Two clocks a listener can feel.
A language model generates autoregressively: one token at a time, each one conditioned on every token before it. That single fact shapes the whole field.
Because words come out one at a time, two timings decide how a reply lands. The first is time to first token, or TTFT: how long after the prompt until the first word appears. It is dominated by the one-time prefill pass, where the model reads the whole prompt. The second is time per output token, or TPOT, also called inter-token latency: how fast each word comes after the first, during the decode phase.
There is a third axis the server cares about: throughput, the total tokens per second produced across all conversations at once. Throughput trades against per-user latency, and that tension runs through everything here. The plain goal: produce the first word quickly, keep the words flowing smoothly, and do it for many conversations on the same hardware. In a spoken setting those clocks decide whether a reply begins inside the window people expect in conversation, a budget the sibling Low-latency voice page owns end to end.
- Time to first token
- Delay until generation produces the first output token.
- Time per output token
- Pace of subsequent decoding steps.
- Throughput
- Total work completed across requests per unit time.
The slow part is usually moving the model, not the math.
For a single reply the bottleneck is memory movement, and the work the model remembers keeps growing. Both shape every technique on this page.
Most of each step is data movement.
To write one token, the machine streams the whole model and its running memory across the chip. For a single reply the time goes to moving that data, not to arithmetic, a point the Databricks engineering guide makes plainly. Adding raw compute does not help much; moving less does.
The model's memory expands with the conversation.
A transformer stores a key and value for every token it has seen so it does not recompute them. That store, the KV cache, grows with context length. Managed naively it gets wasted by fragmentation and duplication, which caps how many conversations fit in memory at once.
The two pull against each other.
Batching more conversations together raises tokens per second but can delay any single user. Sizing the model up improves answers but slows every token. Tuning a serving stack is largely about where to sit on that trade.
Cost is per trip across memory.
Work is per token and per trip across memory, in time and in money. Every technique here is really about getting more value out of each trip: writing more useful tokens, moving fewer bits, or sharing each trip across more conversations.
A serving stack works the problem in layers.
Write more tokens per step, reuse work already done, move fewer bits, and keep the hardware full. These layers stack into a system.
1Write more tokens per step
Speculative decoding.
A small, fast draft model proposes several tokens ahead. The large model then verifies them in a single parallel pass and keeps the longest run that matches what it would have produced anyway. You get several tokens per step with the same output distribution as the large model alone, and no retraining. The foundational result, Leviathan et al. (2023), reports roughly 2x to 3x faster decoding as an industry figure for that work.
A draft-free variant skips the second model entirely. Instead of hosting a separate draft model, Medusa (Cai et al., 2024) adds a few extra prediction heads so the model proposes multiple next tokens itself, then verifies candidates together with tree attention. That removes the cost of training and hosting a second model.
Small-model draft
Target-model verification
Continue from the target model’s correction at the first rejected token.
2Reuse work already done
The KV cache.
The reason the second token is cheaper than the first is reuse. Rather than recomputing the whole context every step, the model keeps the keys and values it already computed for earlier tokens and reads them back. That cache is what makes streaming generation affordable in the first place. How it is stored then decides how many conversations a single server can hold, which is exactly where the next layer comes in.
The mechanics of the cache, and the broader voice budget it sits inside, are picked up in Low-latency voice. Here it is enough to hold one idea: the model remembers its own work, and managing that memory well is half the serving problem.
| Decode step | Reuse | Compute |
|---|---|---|
| Token 1 | Prompt cache | Token 1 K/V |
| Token 2 | Prompt + token 1 | Token 2 K/V |
| Token 3 | Prompt + tokens 1–2 | Token 3 K/V |
3Hold many conversations at once
PagedAttention.
The KV cache grows and shrinks per request, so a naive layout leaves ragged gaps that waste memory. Kwon et al. (2023), the work behind vLLM, stores the cache in fixed-size blocks and pages them like an operating system pages virtual memory. Fragmentation drops to near zero and blocks can be shared across requests, so one server holds many live conversations without waste. The paper reports roughly 2x to 4x higher serving throughput at the same latency as an industry figure for that system. The vLLM project write-up has the memory-layout intuition in pictures.
| Sequence | Logical blocks | Physical blocks |
|---|---|---|
| A | 0 · 1 · 2 | 4 · 9 · 2 |
| B | 0 · 1 | 7 · 1 |
Free finished sequences’ blocks for other requests.
4Move fewer bits
Quantization.
Since decoding is memory-bound, shrinking what gets moved speeds it up directly. Quantization stores the model's weights in lower precision, for example 4 bits instead of 16, using second-order information to keep accuracy close to the original. GPTQ (Frantar et al., 2022) reports end-to-end speedups around 3x to 4.5x over full precision in its own results, with negligible accuracy loss, as an industry figure for that work. Related ideas shrink the KV cache the same way: grouped-query and multi-query attention keep fewer keys and values, so less data moves per token.
- Potential benefit
- Less memory and data movement.
- Check
- Accuracy, quantization overhead, and hardware support.
5Keep the hardware full
Continuous batching.
Instead of running a fixed batch of requests to completion, the server schedules at the granularity of a single token step. It slots new requests in and retires finished ones every iteration, so the hardware stays full and throughput rises sharply, without making any one user wait for a whole batch to finish. The idea comes from Orca (Yu et al., 2022), which introduced iteration-level scheduling and selective batching, the basis for continuous batching in modern servers.
| Request | Step 1 | Step 2 | Step 3 |
|---|---|---|---|
| A | Decode | Decode | Decode |
| B | Finishes | Done | Done |
| C | Waiting | Joins | Decode |
6It composes
Low-latency serving is a system, not one trick.
These layers stack. A quantized model, served with a paged KV cache and continuous batching, generating tokens via speculative decoding, is a single design where the gains compound. The takeaway to carry off this page: no one technique makes a reply fast on its own. They are tuned together, and the result is a serving stack that produces a first word quickly and keeps the words flowing for many conversations at once.
| Technique | Primary target |
|---|---|
| Speculative decoding | Target-model generation work |
| KV cache management | Reuse and memory allocation |
| Quantization | Weight storage and movement |
| Continuous batching | Scheduling and utilization |
Fast inference is what makes responsive assistance practical at scale.
At the field level: a first word that arrives quickly, words that keep flowing, many conversations at once, and headroom for bigger models within the same budget.
For a sales co-pilot, the bridge is at the outcome level. Responsiveness in a live conversation is a research problem the field has been steadily solving, and the techniques above are why a spoken assistant can answer quickly and keep answering smoothly while many people use it at once. The end-to-end voice budget that sits on top of all this is the subject of Low-latency voice.
A first word that arrives quickly.
Work on time to first token is what lets a spoken reply begin inside the window people expect in conversation, instead of a beat too late.
Words that keep flowing.
Steady time per output token is why a long answer does not slow down or stutter as it goes, so a back-and-forth feels natural.
Many conversations at once.
Paged memory and continuous batching are what make responsive assistance practical at scale, rather than a demo for one user. Lower cost per conversation is the same coin.
Headroom for better answers.
When each token costs less to produce, there is room for a larger or smarter model within the same latency budget, so quality and speed stop fighting.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Guessing ahead becomes routine.
Longer draft windows, self-drafting and parallel-head variants, and tree-structured verification are becoming a normal part of the decode loop rather than an add-on.
Split prefill from decode.
Running the one-time prefill stage and the ongoing decode stage on different hardware, so each is tuned for its own bottleneck instead of sharing a compromise.
Fewer bits, quality held.
Toward 4-bit and below for weights, and increasingly for the KV cache itself, while holding answer quality close to the full-precision model.
Share, compress, page.
Sharing common prefixes across conversations, compressing or evicting old context, and paging to keep long sessions cheap as they grow.
Scheduling and decoding together.
Batching, scheduling, and decoding tricks developed as one system rather than bolted on, since the gains compound only when they are designed together.
The papers behind this page.
- Leviathan, Kalman, Matias (2023). Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192.
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). arXiv:2309.06180.
- Frantar, Ashkboos, Hoefler, Alistarh (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323.
- Cai et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774.
- Yu, Jeong, Kim, Kim, Chun (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI 2022.
- Industry context: Databricks, LLM Inference Performance Engineering: Best Practices, and the vLLM project blog on PagedAttention.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
The payoff of all this work is simple: you talk, the reply comes back without a wait, and clean notes are waiting. See it on a real call.
No app required. Just your phone number.