A mixture-of-experts model contains many parallel sub-networks and routes each token to only a few of them. Total parameters grow while per-token computation stays flat. That single decoupling explains almost everything else about MoE, including why it saves compute but not memory.
Picture a serving cluster running a dense 70B model. Every token multiplies against all 70 billion parameters. Now imagine an architecture with 300 billion parameters where each token touches only 40 billion of them, and quality tracks the larger number rather than the smaller one.
That is the trade mixture-of-experts makes. It is genuinely powerful and routinely misunderstood, usually in the direction of expecting a memory saving that never arrives.
What is a mixture-of-experts layer?
Take a standard transformer block. Its feed-forward network is a single dense sub-network every token passes through.
An MoE layer replaces that one feed-forward network with several — the experts — plus a small router network. For each token, the router scores the experts and sends the token to the top k, usually one or two. Outputs are combined, weighted by the router’s scores.
Figure 1 — What happens to one token
- STEP 1Token arrivesSame representation a dense layer would receive
- STEP 2Router scores expertsA small learned network produces one score per expert
- STEP 3Top-k selectedUsually 1 or 2 of 8, 16 or more. The rest stay idle
- STEP 4Selected experts computeOnly these do any work for this token
- STEP 5Weighted combineOutputs merged using router scores, passed onward
The routing decision is per token, not per sequence. Two consecutive tokens in the same sentence routinely go to entirely different experts, which is why “experts” almost never correspond to human-legible topics.
The idea is older than the current wave. Shazeer et al. introduced the sparsely-gated MoE layer in 2017 (arXiv:1701.06538), and the Switch Transformer later simplified routing to a single expert per token to cut communication cost and instability (Fedus, Zoph and Shazeer, arXiv:2101.03961).
What mixture-of-experts is not
Three boundary cases clear up most confusion.
| It is not | Why people think so | What is actually true |
|---|---|---|
| A committee of specialist models | The word “expert” strongly implies domain specialists | Experts are feed-forward blocks inside a layer, not standalone models, and rarely map to human topics |
| A way to reduce memory | Only a fraction of parameters activate per token | Every expert must be resident in memory, because any token might route to any of them |
| An ensemble | Multiple sub-networks contribute to output | Ensembles run all members and average; MoE runs a small subset chosen per token |
The memory point is the one that ruins deployment plans. A model with 47 billion total parameters and roughly 13 billion active per token still needs all 47 billion loaded. You pay memory for the total and compute for the active portion.
Why does the architecture exist at all?
Because dense scaling ties two things together that you might prefer to separate: model capacity and cost per token.
In a dense model, doubling parameters doubles the compute for every token processed, during both training and inference. Capacity is expensive at every step forever.
Sparse activation breaks that link. Capacity grows with total parameter count while cost per token grows only with the active portion. For workloads where knowledge breadth matters more than per-token depth, that is a favourable trade.
Mixtral demonstrated the pattern at accessible scale, using 8 experts per layer with top-2 routing (arXiv:2401.04088), which became the reference design many later open models followed.
What does it cost you?
Four costs, and the memory one dominates practical decisions.
Memory footprint tracks total parameters
Already covered, and worth repeating because it is the most common planning error. MoE is a compute optimisation wearing the costume of a size optimisation.
Load imbalance
Nothing forces the router to distribute tokens evenly. Left alone, it collapses toward a few favoured experts while others go unused, wasting the capacity you paid memory for.
Training therefore adds an auxiliary load-balancing loss that penalises uneven routing. It works, and it competes with the primary objective, so the balance coefficient becomes another thing to tune.
Expert capacity and dropped tokens
Implementations cap how many tokens a single expert will accept in a batch, so buffers stay fixed-size. Tokens exceeding the cap are dropped, passing through unchanged.
Raise the cap and you waste memory on padding. Lower it and you drop more tokens. This tuning has no clean answer and is invisible unless you instrument it.
Communication overhead
In distributed training, experts sit on different devices, so routing means sending tokens across the interconnect and back. On slower fabric this can dominate, which is exactly why Switch Transformer moved to top-1 routing.
Watch for this
Batch composition changes MoE throughput in ways dense models never exhibit. A homogeneous batch — all code, or all one language — concentrates routing on few experts and serialises work that a mixed batch would parallelise. Latency benchmarks run on uniform test data can look excellent and fall apart on real traffic.
How should you reason about the trade?
| Dimension | Dense model | Mixture-of-experts |
|---|---|---|
| Memory required | Proportional to parameters | Proportional to total parameters, not active ones |
| Compute per token | All parameters | Only the selected experts |
| Quality per unit of compute | Lower | Higher, which is the entire point |
| Quality per unit of memory | Higher | Lower |
| Throughput predictability | Stable | Varies with batch composition |
| Fine-tuning difficulty | Well understood | Harder; routing can destabilise |
| Implementation complexity | Low | High, especially distributed |
Read rows one and four together. If you are memory-constrained, a dense model is usually the better buy; if you are compute-constrained or throughput-constrained, MoE is.
Where do engineers meet MoE in practice?
Mostly as consumers rather than builders, which shapes what actually matters.
- Selecting an open model to self-host. Read total parameters for your memory plan and active parameters for your throughput plan. Quoting only one number is how deployments get sized wrong.
- Explaining inconsistent latency. Variance correlated with request type usually points at routing, not at your infrastructure.
- Fine-tuning. Adapter methods interact awkwardly with routing, since which expert sees a token can shift during training.
- Capacity planning. Memory sizing follows total parameters; throughput modelling follows active parameters and batch mix.
What do experienced teams do differently?
They instrument routing from day one.
Logging expert utilisation per batch turns a whole category of mysterious performance problems into a visible chart. Collapse toward a few experts, or systematic dropping under certain traffic, shows up immediately instead of being debugged as a networking issue for a week.
They also benchmark on realistic mixed traffic rather than uniform synthetic loads, because MoE throughput depends on batch heterogeneity in a way dense serving simply does not.
Common misreadings
“Experts specialise by topic”
Interpretability work generally finds routing patterns that are real but not human-legible — closer to token-level syntactic and positional regularities than to subject areas. Do not expect a medicine expert.
“More experts is strictly better”
More experts add capacity and memory, and make balanced routing harder. Returns diminish, and past some point extra experts sit underused while still occupying memory.
“MoE means faster inference”
It means less compute per token. Whether that becomes lower latency depends on memory bandwidth, batch composition and communication overhead. Throughput usually improves; single-request latency often does not.
A short glossary
- Expert
- One of several parallel feed-forward sub-networks inside an MoE layer, only some of which process any given token.
- Router or gate
- A small learned network that scores experts per token and selects the top k.
- Top-k routing
- The policy of sending each token to its k highest-scoring experts, commonly with k of 1 or 2.
- Active parameters
- The subset of parameters actually used to process a single token, as opposed to total parameters held in memory.
- Load-balancing loss
- An auxiliary training objective penalising uneven expert utilisation, preventing routing collapse.
- Expert capacity
- The maximum number of tokens a single expert accepts in a batch, beyond which tokens are dropped.
Key takeaways
- An MoE layer replaces one feed-forward network with several experts plus a router that selects a few per token.
- Total parameters determine memory; active parameters determine compute. These are different numbers and both matter.
- MoE is a compute and throughput optimisation, never a memory optimisation — every expert stays resident.
- Routing is per token, not per sequence, which is why experts rarely correspond to human-legible topics.
- Load-balancing losses exist because routers otherwise collapse onto a few favoured experts.
- Expert capacity limits mean tokens can be silently dropped, so instrument utilisation early.
- If you are memory-bound, buy dense; if you are compute or throughput-bound, buy sparse.
Frequently asked questions
Does mixture-of-experts reduce memory requirements?
No, and this is the most costly misconception. Every expert must be loaded because any token might route to any of them. A model with 47 billion total and roughly 13 billion active parameters still requires memory for all 47 billion. You save compute per token, not storage.
What is the difference between total and active parameters?
Total parameters are everything the model holds and everything you must fit in memory. Active parameters are the subset used to process a single token, determined by how many experts the router selects. Size your hardware on total, and model your throughput on active.
Do experts specialise in particular subjects?
Rarely in any human-legible way. Routing is decided per token rather than per document, and interpretability work generally finds patterns closer to syntactic or positional regularities than to topics. Expecting a dedicated medicine or code expert will mislead your intuitions.
Why do MoE models need a load-balancing loss?
Because unconstrained routers collapse toward a few favoured experts, leaving the rest untrained and unused while still consuming memory. An auxiliary loss penalises uneven utilisation. It competes with the main training objective, so its weighting is a real tuning decision.
What happens when an expert reaches capacity?
Tokens beyond the cap are typically dropped and pass through the layer unchanged, degrading quality silently. Raising capacity wastes memory on padding; lowering it drops more tokens. There is no clean setting, which is why expert utilisation deserves instrumentation.
Is MoE always faster than a dense model?
It uses less compute per token, but latency also depends on memory bandwidth, communication between devices holding different experts, and batch composition. Throughput usually improves on mixed traffic. Single-request latency frequently does not improve at all.
Why does top-1 routing exist if top-2 works better?
Because routing to one expert halves the communication and computation of routing to two, which matters enormously in distributed training. The Switch Transformer adopted top-1 specifically to cut that cost and improve stability, accepting a quality trade for a large efficiency gain.
Is fine-tuning an MoE model harder than a dense one?
Generally yes. Routing decisions can shift during training, so which expert learns from which examples is less stable than a dense model’s uniform updates. Adapter methods interact awkwardly with routing, and load balancing has to be maintained rather than assumed.
References
- Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538, 2017.
- Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
- Jiang, A. Q., et al. Mixtral of Experts. arXiv:2401.04088. Reference design using 8 experts per layer with top-2 routing.
- Lepikhin, D., et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668.
