Load balancing and autoscaling.
Two questions sit under every responsive service. Where should each piece of work go, and how many machines should be running right now. The first is load balancing. The second is autoscaling.
This is an educational walkthrough of that field, in plain language. What the two ideas are, why they matter, how routing and scaling actually work, what they make possible, and where the work is heading. The papers and the system writeups behind each idea are linked, so you can read the source yourself.
One service, many interchangeable workers.
A balancer decides where work goes. An autoscaler decides how much capacity exists. Together they let one logical service feel steady to every user.
A balancer sits in front of a pool of interchangeable workers and routes each incoming request to one of them. The goal is plain: keep every worker usefully busy and no worker overloaded. Autoscaling is the companion idea. It watches how much work is arriving and changes the size of that pool over time, adding capacity when demand climbs and releasing it when demand falls.
There are two axes worth naming. Horizontal scaling changes the number of worker instances. Vertical scaling changes the CPU and memory each instance gets. Real systems do both. A relatable version of the problem: a salesperson is on a live call with a voice co-pilot listening alongside. Each request the co-pilot makes, a transcription, a lookup, a model call, has to land on some worker that is free enough to answer in time. That illustration runs through the page as an example, not as a claim about any specific build.
Demand is bursty, latency is felt at the tail, and workers come and go.
A live call or a fast-updating app is judged by its slow requests, not its average ones. That is what makes routing and scaling worth doing well.
One hot worker is everyone's slow request.
Latency is felt at the tail, not the mean. The question is not whether the fleet is busy on average, it is whether any single worker is hot enough to make someone wait mid-sentence on a live call.
Right at 6am, wrong at 9am.
Demand is uneven through the day. Picture a sales team that all dials at the start of a busy hour. Steady-state capacity that was fine an hour ago is not enough now. Hold peak capacity all day and most of it sits idle.
The fleet changes underneath you.
Workers fail, deploy, and restart constantly. Routing has to keep working as the set of healthy workers shifts, ideally without breaking connections that are already in flight.
Capacity should track real usage.
Holding average capacity drops requests at the peak. Holding peak capacity all day is wasteful. The middle path is to let the fleet size follow demand up and down.
Illustrative uneven queues.
Illustrative distribution of the same 13 jobs.
Same requests, two outcomes. Load balancing is choosing where each request goes, so no one worker becomes everyone's slow request.
How routing and scaling actually work.
Plain language up top, with the named algorithms and systems one layer down. The recurring lesson: very small, very cheap rules often beat clever ones.
1Where should each request go?
Classic routing rules, simplest first.
Round robin hands requests out in rotation and ignores how busy each worker is. Least connections sends the next request to the worker with the fewest in flight, which is better, but naively it needs global state about every worker. The result worth featuring sits between them: sample two workers at random and send the request to the less loaded of the two. That small change yields an exponential improvement in the maximum load compared with a single random choice, at almost no coordination cost, as shown by Mitzenmacher (2001).
For very large fleets, Join-Idle-Queue (Lu et al., 2011) refines the idea: instead of dispatchers polling workers at request time, idle workers advertise themselves, so routing to a free worker costs no request-time communication.
It has less queued work than the other sampled worker, A.
2A ladder of sophistication
More information, better decisions, more coordination.
The honest tradeoff runs the length of this ladder. Knowing more about each worker makes for better routing, but gathering that knowledge costs coordination. The two-choices rule is interesting precisely because it buys most of the benefit of full knowledge for the price of two random lookups.
| Rule | Decision signal |
|---|---|
| Round robin | Next worker in rotation |
| Least connections | Current in-flight connection counts |
| Two choices | Load of two sampled workers |
| Join-Idle-Queue | Advertisements from idle workers |
3Keeping connections stable
Do not reshuffle everyone when one worker leaves.
When a worker is added or removed, you do not want every existing connection reassigned. Consistent hashing and connection tracking keep most flows pinned to the same backend across membership changes, so only a small share of connections move. Maglev (Eisenbud et al., 2016) is a real software load balancer built on exactly this combination, fronted by equal-cost multipath (ECMP) routing from the network. A balancer also continuously probes workers with health checks and routes around the ones that are slow or down, so a single bad machine does not become everyone's slow request.
| Key group | Before | After B leaves |
|---|---|---|
| Group 1 | Worker A | Worker A |
| Group 2 | Worker B | Next eligible worker |
| Group 3 | Worker C | Worker C |
Illustrative consistent-hash ownership.
4How many workers should exist?
The autoscaling control loop.
Measure a signal, CPU, queue depth, request rate, or latency, compare it to a target, and add or remove capacity to close the gap. Horizontal autoscalers change instance counts. Vertical autoscalers change per-instance limits. Autopilot (Rzadca et al., 2020) does both automatically and reports large reductions in wasted, over-requested capacity. Underneath sits a cluster manager that decides which machine each worker lands on, packs work efficiently, and reschedules around failures. Borg (Verma et al., 2015) is the canonical described system of that kind.
- 01Measure
Observe utilization or queue length.
- 02Compare
Check against the target and policy.
- 03Adjust
Change capacity and wait for readiness.
- Solid blue
- Workload demand.
- Dashed slate
- Provisioned capacity, responding after a delay.
5Tradeoffs to name honestly
Every choice here costs something.
More information makes better routing decisions but costs coordination. Scaling up takes time, so an autoscaler has to react before the queue builds, not after. Aggressive scale-down saves money but risks a cold fleet when the next burst arrives. None of these have a single right answer. The discipline is choosing where on each tradeoff a given service should sit, given how it is used.
- Scale-up delay
- Keep enough headroom for workers to become ready.
- Scale-down stability
- Avoid repeatedly removing and recreating capacity.
- Routing information
- Balance better decisions against coordination overhead.
What good routing and scaling make possible.
At the field level: steady response times under uneven demand, cost that tracks real usage, and graceful behavior during failures and deploys.
Bring the running examples back, as illustrations of the field rather than claims about any build. A voice co-pilot is useful when its requests during a call land on workers that can answer in time. A fast-updating app stays responsive when fresh data has to reach the screen quickly and many people are watching at once. Routing and scaling are what keep a latency-sensitive experience steady under a burst.
Steady under uneven demand.
Response times hold through the day even as load rises and falls, which is what makes a live experience feel reliable rather than occasionally laggy.
Cost that follows usage.
Capacity tracks real demand up and down instead of being provisioned for the peak around the clock.
Graceful through failures.
Routing flows around unhealthy workers and scaling refills the gap, so deploys and restarts do not become everyone's slow request.
Fresh data, fast.
For a co-pilot on a call, the value is getting a lookup or a model call answered in time. That is an example of the problem, not a description of our plumbing.
This page describes the field. We stay neutral on how any specific service, including ours, routes and scales. For the product itself, the one line we use: our own model, built for sales conversations.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
Scale ahead of a known surge.
Forecasting demand to add capacity before a recurring peak, such as the start of a busy hour, rather than purely reacting once the queue starts to build.
Shrink the lag to serving.
Lighter isolation and quicker cold starts narrow the gap between "we need capacity" and "capacity is serving," which is the hard part of reacting to bursts.
Not just free, but near and warm.
Sending work to a worker that is close or already warm, not only one that is idle, which matters most for interactive, real-time workloads.
One problem, not two loops.
Treating routing and scaling together so the balancer and the autoscaler do not work against each other, instead of tuning two independent loops.
The papers behind this page.
- Mitzenmacher (2001). The Power of Two Choices in Randomized Load Balancing. IEEE Transactions on Parallel and Distributed Systems. doi:10.1109/71.963420.
- Lu, Xie, Kliot, Geller, Larus, Greenberg (2011). Join-Idle-Queue: A Novel Load Balancing Algorithm for Dynamically Scalable Web Services. Performance Evaluation.
- Eisenbud et al. (2016). Maglev: A Fast and Reliable Software Network Load Balancer. USENIX NSDI 2016.
- Rzadca et al. (2020). Autopilot: workload autoscaling at Google. EuroSys 2020. doi:10.1145/3342195.3387524.
- Verma et al. (2015). Large-scale cluster management at Google with Borg. EuroSys 2015. doi:10.1145/2741948.2741964.
Keep reading.
Sibling explainers in the Research section. Routing and scaling are part of the latency story.
Talk. Done.
The systems are deep, but the promise is plain: you talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.