Mixture of experts.
A model can be large in what it knows yet small in what it runs per input. Here is the architecture that makes that possible, in plain language.
A mixture of experts is a model built from many smaller sub-networks, with a small router that sends each input to just a few of them. The rest sit idle. So total capacity can grow while the work done per input stays roughly flat. We walk it the way the field is built: what it is, why it matters and why it is hard, how it works, what it unlocks, and where it is heading. The papers behind each idea are linked, so you can read the source yourself.
Many specialists, a small router, a few chosen per input.
Think about how a large company answers questions. A contract goes to legal, a billing question goes to finance, a technical question goes to support.
You do not send every question to every employee. Most of the staff is available, but only the right few work on any one request. The organization can be large and broadly capable, yet each question only ever costs a few people's time.
A mixture of experts model works the same way. Instead of one dense network where every part runs on every input, the model is built from many smaller sub-networks called experts. A small router, also called a gate, looks at each input and sends it to just a few experts. The others sit idle for that input. The field calls this sparse activation: many parameters exist, only a few are used at a time. The decision of which experts run depends on the input, which is why it is also called conditional computation.
The distinction the whole page hangs on is total parameters, how much the model can hold, versus active parameters, how much it actually runs per input. In a dense model these are the same number. In a sparse mixture of experts, total can be far larger than active.
The payoff is real, and so is the difficulty.
Sparsity separates capacity from compute. But choosing which experts run, and keeping them all useful, turns out to be the hard part.
Capacity and compute, pulled apart.
Quality tends to improve as you add capacity, but a fully dense model of that size makes every input more expensive to run. A sparse mixture lets total capacity grow while per-input compute stays roughly flat, because each input only touches a slice of the network.
The router and the experts learn at once.
The router has to learn which experts suit which inputs while the experts are themselves still learning. The decision is discrete, pick expert A and not B, which is harder to train through than a smooth dense layer.
Load balancing.
Left alone, routers tend to crowd onto a few favorite experts while the rest go unused, which wastes capacity. Training needs explicit pressure, an auxiliary load-balancing loss and a limit on how much any one expert will take, to keep work spread out.
Moving data between experts.
Experts often live on different accelerators, so routing an input to its chosen expert means moving data between devices, an all-to-all exchange. The experts also occupy memory even when idle. Sparsity saves compute per input but adds a coordination and footprint cost.
Large sparse models can be unstable.
Switch Transformer reports specific techniques, selective precision, careful initialization, and router tweaks, needed to train large sparse models reliably. The architecture buys efficiency, not a free lunch.
Two numbers, not one.
The headline idea is worth holding clearly. A model can hold far more parameters than it runs at once. That gap is the whole point, and it is what the rest of this page is about.
- Total capacity
- All eight experts’ parameters are stored.
- Active expert work
- Two experts run for this illustrative token.
How it works, one mechanism at a time.
Replace one big block with several. Add a router. Then face the practical problems honestly: balancing, capacity, and moving data across hardware.
1The expert layer and the router
Several parallel sub-networks, and a gate that picks.
The modern form drops the idea into the feed-forward layers of a transformer. Replace one big feed-forward block with several parallel ones, the experts. Each is a full sub-network, and only the chosen few run per token. A small learned function, the router or gate, scores the experts for the current input and selects the top few, top-k. Their outputs are combined, weighted by the router's scores. This is the layer introduced in Shazeer et al. (2017).
Picking k is the central knob. More experts per token means more compute and usually a bit more quality. Fewer is cheaper. The same structure goes back to Jacobs et al. (1991), where a gating network learned to split a task into sub-tasks across several small experts.
Top-1 routing
One selected expert for this token.
Top-2 routing
Two selected experts; combine their outputs.
2The simplest form first
Top-1 routing, then the real-world caveats.
A clean way to teach the architecture is to simplify the router all the way down. Switch Transformer (Fedus, Zoph, Shazeer, 2021) sends each token to exactly one expert, which cuts routing and communication cost and makes the system easier to reason about. The rest of the work is then the practical problems that creates: instability, balancing, and the cost of moving data.
This is the useful pattern across the field. Introduce the simplest version, get it working, then add the real-world handling one piece at a time rather than all at once.
Token representation
The selected expert transforms this token representation.
3Keeping every expert useful
Load balancing and expert capacity.
This is the central practical difficulty, named honestly. An auxiliary loss rewards spreading tokens evenly across experts. A capacity factor caps how many tokens any one expert will take per batch, dropping or rerouting the overflow. Together they stop the router from collapsing onto a few favorites. The treatment in Switch Transformer and the routing in GShard (Lepikhin et al., 2020) are the reference points.
Without this pressure, capacity is wasted: a model can hold many experts but only ever use a handful, which defeats the purpose of building them.
Concentrated routing
More balanced routing
4When the model outgrows one device
Sharding experts across hardware.
When the model is too big for one accelerator, the experts are spread across many, and the framework handles routing each token to wherever its expert lives. GShard (Lepikhin et al., 2020) introduced expert sharding and the all-to-all exchange that makes very large sparse models trainable.
The travel itself is a cost. A token routed to an expert on another device has to make the trip there and back, which is why communication shows up so often in the engineering of these systems.
| Device | Hosted experts |
|---|---|
| Device 1 | A · B |
| Device 2 | C · D |
| Device 3 | E · F |
- Router selects D
The token representation is on device 1.
- Dispatch
Transfer it to device 2 for expert D.
- Return
Combine the expert result in the model’s computation.
5A worked modern example
Eight experts per layer, two active per token.
To make the total-versus-active distinction concrete, Mixtral (Jiang et al., 2024) places eight experts in each layer and routes each token to two of them. The model holds a large total parameter count but only activates a fraction per token. We cite the architecture as a clear illustration of the idea. We do not rank it against anything.
Read alongside the help-desk picture at the top of the page, the numbers land plainly. Many experts exist in total. Only two do the work for any given token.
6Where this sits in the stack
An architecture choice underneath the serving layer.
Mixture of experts is a choice about how the model is built. It sits underneath the techniques that decide how fast a model serves and generates tokens. Those belong to a sibling page, not this one. For the serving and token-generation side, see Low-latency inference. We cross-link rather than duplicate.
It is also worth keeping two ideas separate. Which expert runs is a compute-efficiency choice. Which data a person may see is an access-control and retrieval concern, and it lives on the security and retrieval pages. They are not the same thing, and a router does not provide data isolation.
Expert routing
- Controls
- Which neural sub-network processes a token.
Access policy
- Controls
- Which records may enter the request’s context.
What it unlocks, in plain terms.
More capacity at roughly flat per-input cost, faster training to a given quality, a natural home for specialization, and a lever for scale.
As an example of the idea, and not a claim about any product: a single broad model could, in principle, develop different internal specialists. Some good at the language of one industry, some at another way of selling, some at a particular kind of question. Routing is what lets a model carry that breadth without paying for all of it on every request. Mixture of experts is one architecture the field uses to make that affordable. This is field context, not a description of how any product on this site is built.
Capacity without proportional cost.
The headline benefit: a model can know more without each input costing proportionally more to run.
Faster training to a quality.
The source papers report several-times training speedups over dense models of comparable quality for the same compute budget. The figure belongs to those papers, not to anyone reading this page.
A home for specialization.
Experts can, in effect, divide a broad problem into parts. That was the original 1991 motivation, and it is still the intuitive selling point.
A lever for scale.
Sharding experts across hardware is how very large published models reach their parameter counts while staying trainable.
This explainer covers published research. To see how AI REV Voice helps with sales conversations, explore Prism.
The direction the field is heading.
We describe the direction, not a roadmap of our own.
More stable ways to choose.
Routing research continues toward steadier, better-balanced gates, including expert-choice routing, where experts pick tokens rather than tokens picking experts, and finer-grained experts.
Inference has its own problems.
Because only a few experts run per token, serving a sparse model means loading the right experts and handling the straggler effect when one is overloaded. This links to low-latency inference.
Smaller experts, some always on.
Splitting capacity into many small experts, sometimes with a few always-on shared experts, to get more specialization per parameter.
The same idea, elsewhere.
The conditional-computation idea appears in vision and multimodal models, and in parameter-efficient fine-tuning, where routing adapts a base model to a setting.
Huge-model quality at small-model cost.
The field still works on getting the quality of a large dense model at the cost of a small one, reliably, without the balancing and communication overhead eating the savings.
The papers behind this page.
- Jacobs, Jordan, Nowlan, Hinton (1991). Adaptive Mixtures of Local Experts. Neural Computation 3(1).
- Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.
- Lepikhin et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668.
- Fedus, Zoph, Shazeer (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
- Jiang et al. (2024). Mixtral of Experts. arXiv:2401.04088.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
The architecture is the field's. The promise is ours: you talk, and clean, complete notes are waiting. See it on a real call.
No app required. Just your phone number.