Caching, from CPU to CDN.
Almost everything that feels instant feels that way for one reason: the data it needed was already close, not freshly fetched. That trick has a name, and it shows up at every scale of a computer system. Here is the field, in plain language.
A cache is a small, fast copy of something kept close to where it is needed, so the slow original is fetched less often. That is the whole idea, and it repeats from the processor up to the edge of the network. The reason it exists is unglamorous: fast storage is small and costly, slow storage is big and cheap, and the gap between how fast a processor wants data and how fast memory can hand it over has widened for decades. We walk it the way the field is built: what a cache is, why it matters, how it works, what it unlocks, and where it is heading. The papers and the spec behind each idea are linked, so you can read the source yourself.
A small, fast copy, kept close.
That one sentence is the whole field. The vocabulary fits in three words, and the trick is the same at every scale.
When a request finds its copy already in the cache, that is a hit. When it does not, that is a miss, and the system has to go fetch the slow original. The fraction of requests served from the copy is the hit rate, and it is the number that decides whether a cache is earning its keep. A cache with a low hit rate is just overhead.
The thing to watch for, as this page climbs from the processor to the edge of the network, is that it is the same idea repeated. Picture a live sales call. The rep asks the co-pilot a quick question, and the answer is only useful if it lands while the prospect is still talking. Behind any system that feels that quick is a stack of caches, each one a fast copy of the rung below it. We explain the field. We make no claim about how any one product is built.
Fetch the current result, then save a copy.
Use the saved copy while it is still valid.
Fast, big, cheap: pick two.
Fast storage is small and costly. Big storage is slow and cheap. You cannot have all three, so systems are built as a ladder of stores.
The gap is also widening. Processor speed outran memory speed for decades, and Wulf and McKee (1995) named the result the memory wall: without caching, a fast processor spends most of its time waiting on memory. Caching is the main trick the whole field uses to hide it.
Latency is layered. A value already on the chip arrives in roughly a nanosecond. Main memory is tens to a hundred times slower than that. A disk seek or a network round trip is slower again by orders of magnitude. Every layer of cache exists to avoid paying the cost of the next layer down. The figures beside this paragraph are well-known orders of magnitude from the field, not any product's numbers.
- CPU cache
- Very small, very close to computation.
- RAM
- More capacity, with a slower lookup than on-chip cache.
- Local storage
- Persistent, larger, and slower again.
- Remote origin
- Adds a network round trip.
The same idea, at every scale.
Walk the ladder from the processor to the edge, then meet the two hard problems the whole field keeps wrestling with: what to throw out, and how to stay honest.
1The hierarchy, CPU to CDN
Each rung is a cache for the rung below it.
The processor keeps the last few values in tiny on-chip caches (L1, L2, L3). Below that sit main memory and the operating system page cache. Above the database, applications keep hot records in fast in-memory stores so they do not hit the database for every read, the role that Memcached and Redis play. The database keeps its own buffer cache. Out at the edge, the browser cache and a content delivery network keep copies near the user. Same idea, repeated: keep a fast copy close.
It works because of locality. Programs tend to reuse the same data soon (temporal locality) and to use data near what they just touched (spatial locality). Caches bet on both. No locality, no benefit.
- Browser or edge
Use a fresh response close to the reader.
- Application cache
Reuse a recently computed value.
- Database or origin
Fetch the authoritative record on a miss.
2Eviction
What do you throw out?
A cache is smaller than the thing it fronts, so when it fills up it has to drop something to make room. The common default is Least Recently Used (LRU): evict whatever has gone untouched longest, on the bet that recent use predicts reuse. The field has a long menu beyond it, mapped by Podlipnig and Böszörményi (2003). Two worth naming: LFU (least frequently used) keeps the most-requested items, and TTL (time-to-live) simply expires an entry on a timer.
The choice is a bet about the future from the past. A policy that guesses well keeps the hit rate high. A policy that guesses badly evicts the very thing about to be asked for again.
| Policy | Eviction rule |
|---|---|
| LRU | Remove the least recently used item. |
| LFU | Remove a least frequently used item. |
| TTL | Expire an item after its time limit. |
Example: A was read most recently; B has the most lifetime reads. LRU and LFU can therefore keep different entries.
3Write strategies
Write-through, or write-back.
When data changes, a write-through cache updates the fast copy and the source of truth together: simple and safe, but every write pays the slow cost. A write-back cache updates the fast copy now and the source of truth later: faster writes, but the copy and the original disagree for a window, and a crash in that window can lose the lagging write.
This is the classic speed-versus-safety dial, and it is the same dial at every layer, from a CPU cache to an application cache. The page presents the trade-off, not a recommendation. Which way to turn the dial depends on what a given system can afford to lose.
Write-through
- Update
Write the origin and refresh the cached value.
- Acknowledge
Wait for the required writes.
Write-back
- Update cache
Mark the value dirty and acknowledge.
- Flush later
Persist the change to the origin.
4Invalidation
Keeping copies honest.
"There are only two hard things in computer science: cache invalidation and naming things" is a field joke because it is true. A cache is a copy, and copies go stale. When the original changes, every stale copy has to be found and refreshed or thrown out.
On the web, the protocol does this with two ideas, written down in RFC 9111. Freshness: a copy is good until it expires. Validation: once it expires, the cache asks the origin whether anything changed, using an ETag or a Last-Modified date, and reuses the copy if the answer is no. That is how a just-changed record can reach the app fast and correctly, fresh where it counts, without re-fetching everything.
- Fresh copy
Serve it according to the freshness policy.
- Stale copy
Send If-None-Match with its stored ETag.
- Origin responds
304 means reuse the body; a changed representation returns a new body and validator.
5Spreading across machines
Consistent hashing.
One server cannot hold everything, so the keyspace is split across many. The naive way, hash a key and take it modulo the number of servers, reshuffles almost every key whenever a server is added or removed, which empties the caches at the worst possible moment.
Karger et al. (1997) place servers and keys on a ring instead, so adding or removing one server moves only a small slice of the keys and the rest stay put. This is the foundation under distributed caches and, at internet scale, under content delivery networks routing a user to the right nearby copy.
Modulo routing
server = hash(key) % NChanging N can change many key assignments.
Consistent hashing
- Map keys and servers
Place both on a logical ring.
- Remove one server
Its assigned keys move to the next eligible server.
6At internet scale
A cache for the whole network, and the hard part nobody expects.
A content delivery network keeps copies of content in many locations near users, so a request is served from a nearby city instead of a distant origin. It is the same locality bet, played at the scale of the planet, and it leans on consistent hashing to find the right nearby copy. Nygren, Sitaraman, and Sun (2010) give a clear account of one such network.
Put all of this in one production system and the genuinely hard engineering turns out not to be the cache itself. Nishtala et al. (2013) describe a large Memcached deployment fronting a read-heavy service, where the real work is keeping the cache consistent with the database under heavy change. The mechanism is easy. Staying honest is the job.
- Reader
Requests a document or asset.
- Nearby edge
Returns a fresh cached copy on a hit.
- Origin on a miss
Supplies the representation for the edge to store and return.
Why any of this matters above the plumbing.
At the field level: speed that feels instant, scale that does not melt the database, lower cost and load, and a recent change in front of a user fast.
For a sales co-pilot, the bridge is at the outcome level, as an example of the idea and not a description of any system. On a live call, the value of an answer is gone if it arrives after the moment. Keeping the data the moment needs close by is how systems in this field stay quick, and good invalidation is how a just-changed record reaches the app fresh where it counts. None of this describes our plumbing.
Speed that feels instant.
A page, an app, or a spoken answer lands fast usually because the data was already close, not freshly fetched across the slowest layer.
Scale without melting the source.
A cache absorbs the flood of repeated reads so the slow source of truth only sees the misses. That is what lets a service hold up as users climb.
Fresh where it counts.
Caching plus good invalidation gets a recent update in front of a user quickly, the live-call case where a just-changed record needs to reach the app fast and correctly.
Resilience, and lower load.
Serving a copy is cheaper than recomputing the original, and a warm cache can keep serving for a while even when the origin is slow or briefly down.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Learned and adaptive policies.
Beyond LRU and LFU, policies that try to predict what will be reused. The survey above stays the baseline these are measured against.
Reusing computed results.
Not redoing work a model has already done: the key/value cache inside a transformer, covered on Low-latency voice and Inference optimization, and the caching of retrieved context and embeddings on Embeddings and vector search.
Logic, not just files, near the user.
Content delivery networks moving from caching static files to running logic at the edge, pushing both data and computation closer to where the request is.
Staying current, not just close.
As more data updates in real time, the hard part shifts from storing copies to invalidating them fast and correctly. The newest work is mostly about staying fresh, not about going faster.
The sources behind this page.
Five papers and one canonical spec.
- Wulf and McKee (1995). Hitting the Memory Wall: Implications of the Obvious. ACM SIGARCH Computer Architecture News.
- Karger, Lehman, Leighton, Panigrahy, Levine, Lewin (1997). Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web. ACM STOC.
- Podlipnig and Böszörményi (2003). A Survey of Web Cache Replacement Strategies. ACM Computing Surveys.
- Nygren, Sitaraman, Sun (2010). The Akamai Network: A Platform for High-Performance Internet Applications. ACM SIGOPS Operating Systems Review.
- Nishtala et al. (2013). Scaling Memcache at Facebook. USENIX NSDI.
- IETF (2022). RFC 9111: HTTP Caching. Internet Engineering Task Force.
Keep reading.
Sibling explainers in the Research section, on latency and systems.
Talk. Done.
Speed and freshness are research problems this field has been steadily solving, and they are part of why a co-pilot can answer like a person. See it on a real call.
No app required. Just your phone number.