Mixture of Experts, Explained: Why the Biggest AI Models Activate Only Part of Themselves

For most of deep learning's history, a language model had one dial that mattered: size. More parameters meant more knowledge, and running the model meant touching every parameter for every token. That bargain built GPT-3, PaLM and a generation of large dense transformers, and it also built a wall. Past a few hundred billion parameters, the compute and memory needed to generate each token rise with the parameter count, and inference cost starts to dwarf training cost.

Related reading: Small Language Models Are Having a Moment · OpenAI Revenue Below Expectations Sends AI-Linked Stocks Lower.

Sparse mixture-of-experts (MoE) is the main way the field has climbed over that wall. In an MoE model, each transformer layer's single feed-forward network is replaced by many smaller feed-forward networks, called experts, and a small router that sends each token to only a few of them. The model can hold hundreds of billions or trillions of parameters while doing the arithmetic of a much smaller one. That choice has consequences for memory, load balance and serving cost, and those consequences explain why MoE is now standard at the frontier.

Dense versus sparse: the tradeoff in one paragraph

A dense model uses all of its parameters on every token. Feed a 70-billion-parameter dense model one token and it runs 70 billion parameters of matrix math to produce the next one. Capacity and compute are the same thing.

A sparse MoE model breaks that link. The attention layers stay dense and shared. The feed-forward layers, which hold most of a transformer's parameters, become a set of experts. The router decides, per token and per layer, which experts run. Total parameters measure the model's capacity, while activated (or active) parameters measure what it costs to process one token. MoE lets the first number grow much faster than the second.

Two terms are worth fixing up front. An expert is a small feed-forward network, usually with the same architecture as the dense block it replaces, but with its own weights. Top-k routing means the router keeps the k highest-scoring experts for each token: top-1 sends a token to one expert, top-2 to two, and so on. The router's scores also weight how the selected experts' outputs are combined.

A worked example: Mixtral 8x7B

Mistral AI's Mixtral 8x7B, released in December 2023, made the idea concrete for open-weight models. Each layer holds 8 expert feed-forward blocks. For every token, a router picks the top 2. The model has 46.7 billion total parameters but runs only 12.9 billion of them per token, about 28 percent. Mistral reported that it matched or beat Llama 2 70B on most benchmarks while using roughly 5x fewer active parameters at inference.

The name is a little misleading. "8x7B" does not mean eight independent 7B models. The routers, attention layers and embeddings are shared across experts, so the total is 46.7B rather than 56B, and the active count is 12.9B rather than 14B.

DeepSeek-V3, released in December 2024, pushes the same idea further. Each MoE layer has 256 routed experts plus 1 shared expert that always runs. For each token, the model activates the shared expert and the top 8 routed experts. That gives 671 billion total parameters and 37 billion activated per token, roughly 5.5 percent. The fine-grained split, introduced in the DeepSeekMoE paper, gives the router many more ways to combine expertise than a handful of large experts would, and the shared expert handles common patterns so the routed experts do not each have to relearn them.

When an expert overflows: capacity factor and token dropping

Routing is data-dependent, and that creates a scheduling problem. A batch of tokens does not split evenly across experts. If one expert is popular, it can receive far more tokens than the others.

Production MoE layers handle this with a fixed buffer per expert. Each expert is allotted a capacity equal to (tokens per batch / number of experts) multiplied by a capacity factor. Suppose a batch has 4,096 tokens, the layer has 8 experts and routing is top-2. On average each expert would receive (4,096 × 2) / 8 = 1,024 tokens. With a capacity factor of 1.0, 1,024 is the buffer; if 1,300 tokens arrive at one expert, the extra 276 are dropped at that layer and pass through the residual connection unchanged. A capacity factor of 1.25 raises the buffer to 1,280 and reduces dropping, but the padded, unused slots still consume compute and communication.

Dropping tokens is a real cost, not a cosmetic one. Rare tokens and under-represented languages are the most likely to be affected, because the router is least confident about them. Google's Switch Transformer paper, published in January 2021, treats the capacity factor as a first-class tuning knob for exactly this reason.

The balancing act

Left alone, a router tends to collapse: it discovers a few experts early and sends most tokens to them, leaving the rest under-trained. The standard fix is an auxiliary loss that penalizes uneven load, added to the training objective. The problem is that pushing too hard on balance can hurt quality, because it overrides what the router has learned.

DeepSeek-V3 introduced an alternative it calls auxiliary-loss-free balancing. Instead of a penalty, each expert gets a bias term that is added only when scoring routes. During training the system watches each expert's load and nudges the bias up or down. Overloaded experts get a lower bias, underloaded ones a higher one, and the gate value actually used to weight the output stays true to the router's original score. The report says this kept expert loads balanced while avoiding the quality hit of a large auxiliary loss.

A short history, with numbers

The idea predates the transformer. In 1991, Robert Jacobs and colleagues proposed adaptive mixtures of local experts, where a gating network learns to combine the outputs of several specialist networks. The modern version arrived with a 2017 Google paper, Outrageously Large Neural Networks, which added sparsity and noisy top-k gating and trained an MoE layer with up to 137 billion parameters between LSTM layers.

Two Google papers then turned the idea into infrastructure. GShard, submitted in June 2020, pushed a sparsely-gated MoE translation model past 600 billion parameters and trained it on 2,048 TPU v3 chips in four days across 100 languages. Switch Transformer, posted in January 2021, simplified routing to a single expert per token and showed that top-1 could work. Its largest variant, Switch-C, reached 1.6 trillion parameters, and the authors reported a 4x pre-training speedup over the dense T5-XXL at matched compute.

Open weights changed the stakes. Mixtral 8x7B shipped in December 2023 under Apache 2.0 and became a reference point for a generation of open MoE models. DeepSeek's V2 and V3 refined the architecture with fine-grained and shared experts. By 2025 the approach had reached the largest labs: Google confirmed that Gemini 1.5 Pro is a sparse MoE model, and Meta's Llama 4 Scout and Maverick, announced in April 2025, are MoE models with 17 billion active parameters against 109 billion and 400 billion total. OpenAI has never confirmed GPT-4's architecture, but a widely cited 2023 report described it as a mixture of experts, and the pattern has held across nearly every subsequent frontier release.

“This technique increases the number of parameters of a model while controlling cost and latency, as the model only uses a fraction of the total set of parameters per token.” — Mistral AI, on Mixtral 8x7B

Why labs choose sparse models

The appeal is arithmetic. Dense scaling raises quality and cost together. Sparse scaling raises the parameter count, which is the reservoir of knowledge, while holding the per-token compute roughly fixed. That is a better exchange rate for models that will serve billions of tokens.

Training benefits too. DeepSeek-V3's technical report says the full model, trained on 14.8 trillion tokens, used 2.788 million H800 GPU-hours, a figure that would be far higher for a dense model of comparable measured quality. At the frontier, the decision is less about beating dense on any single benchmark and more about reaching a given level of quality for a sustainable price.

Serving is the hard part

MoE saves compute, but it does not save memory, and serving is where that asymmetry bites.

Every expert must be resident somewhere, even though only a few run for any token. Serving memory therefore scales with the total parameter count, not the active count. Mistral notes in its Mixtral paper that serving memory is proportional to the 47-billion sparse parameter count, while inference compute tracks the 13 billion active count. For DeepSeek-V3 that means holding 671 billion parameters of weights to run 37 billion per token. On a data center full of GPUs, that is a large amount of expensive memory idling.

Distributing experts across GPUs, a strategy called expert parallelism, adds communication. Tokens routed to an expert on another device must be sent there and their results sent back, and systems often use all-to-all communication to do it. If one expert is hot, other GPUs wait for it. One study of MoE serving measured GPU time lost to this straggler effect at up to 70 percent in skewed workloads.

Batching, normally the main lever for inference efficiency, works differently for MoE. A small batch can activate almost every expert while giving each one very few tokens, which destroys the arithmetic intensity the hardware needs. A 2024 paper on MoE inference observed that Mixtral 8x7B, with 8 experts per layer and top-2 routing, activated 7.63 of 8 experts on average on a chatbot workload, because different tokens chose different experts. Serving systems respond with routing-aware batching, expert caching and offloading, and by treating the prefill phase, where many tokens run at once, differently from the decode phase, where one token is generated at a time.

Quality and interpretation questions

The architecture also shapes how a model behaves, in ways researchers are still working out.

Routing is not stable under small changes. A token that sits near a boundary between two experts can flip to a different expert with a tiny change in its representation, and that flip changes which weights process it. Fine-tuning can therefore redistribute routing in ways that are hard to predict. Misrouting a rare or unfamiliar input is the case that matters most, and it is the same case where token dropping is likeliest.

Experts also do not map cleanly onto human-legible topics. Early work found experts specializing along syntactic and semantic lines, and later models show domain-level skews, but a single expert typically carries a mix of patterns. That makes an MoE model harder to interpret and harder to edit: to change a behavior you may need to know which experts fire and how, which is not visible in the weights alone.

Trade-offs and what to watch

Sparse mixture-of-experts is not a free upgrade. It trades compute for memory and for operational complexity, and it asks training to solve a load-balancing problem that dense models never face. Dense models remain the better choice for small models, for latency-sensitive single-stream use, and for settings where simplicity and predictability matter more than peak quality.

The direction of travel is clear. Experts are getting finer-grained, shared experts are becoming standard, balancing is moving away from blunt auxiliary losses, and the serving stack is being rebuilt around the fact that a model now has many parts that come and go. The open questions are practical: how to serve trillions of parameters without leaving most of the memory idle, how to keep routing stable enough to trust, and whether the gains from sparsity keep pace as the largest models grow.

Cover image: NVIDIA DGX B200 HGX eight-way GPU baseboard, photographed by Pokiiri, sourced from Wikimedia Commons under a CC BY-SA 4.0 license.