Mixture of Experts: Why Conditional Computation Is the Path to Trillion-Parameter Models

Training a dense 1-trillion-parameter model would require roughly 25 million GPU-hours — about $75M at current H100 pricing. MoE models achieve the same quality with 5-10x less compute by activating only a fraction of parameters per token. DeepSeek V3 (671B total, 37B activated) outperforms Llama 3 405B (dense) while training for a fraction of the cost.

This post covers the full MoE stack: router design, load balancing, fine-grained experts, expert parallelism systems, and the production serving challenges.

Why MoE: The Scaling Law Motivation

The Chinchilla scaling law shows that model quality improves predictably with both parameter count and training tokens. But FLOPs scale linearly with activated parameters — not total parameters. MoE exploits this gap:

$\text{FLOPs per token} \propto P_{\text{activated}}, \quad \text{Quality} \propto P_{\text{total}}^{0.5} \times D^{0.5}$

A 671B MoE model activating 37B per token costs the same FLOPs-per-token as a 37B dense model, but has the representational capacity of a much larger model. The quality comes from having more specialized parameters to choose from, even though each token only uses a small fraction.

Router Design: How Tokens Choose Experts

The router (also called gating network) is a small linear layer that takes a token’s hidden state $h$ and produces logits over all experts:

$g(h) = \text{softmax}(W_g \cdot h + b)$

The top-$k$ experts by logit value are selected, and their outputs are weighted by the corresponding gate values.

Top-K Routing

The standard approach. Each token picks its top-$k$ experts:

def top_k_routing(hidden, router_weight, k=8):
    logits = hidden @ router_weight.T           # [batch, num_experts]
    topk_vals, topk_ids = logits.topk(k, dim=-1)  # [batch, k]
    gates = F.softmax(topk_vals, dim=-1)         # Normalize over selected experts
    return topk_ids, gates

Problem: Popular experts get overwhelmed with tokens (load imbalance), while unpopular experts are underutilized.

Expert Choice Routing

Inversion: instead of tokens choosing experts, experts choose tokens. Each expert picks its top-$C$ tokens from the batch. This guarantees perfect load balance by construction.

Problem: Some tokens may not be selected by any expert (token dropping) or selected by too many (compute waste).

Group-Limited Gating (DeepSeek V3)

A hybrid: tokens choose experts, but with a constraint — no expert can receive more than a group capacity limit within a group of tokens. This approximates load balance without the rigidity of expert choice.

Load Balancing: The Central Challenge

If all tokens route to the same 2 experts, the other 254 sit idle. Throughput drops to single-expert speed. Load balancing is the most important MoE systems challenge.

Auxiliary Loss Approach (Switch Transformer, GShard)

Add a differentiable loss term that penalizes imbalanced routing:

$L_{\text{balance}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot P_i$

where $f_i$ is the fraction of tokens routed to expert $i$ and $P_i$ is the average router probability for expert $i$. This loss pushes the router toward uniform distribution.

The problem: This loss directly competes with the language modeling objective. The router must balance between “send this token to the expert that would process it best” and “send it somewhere underutilized.” This tension reduces final model quality by 0.1-0.3% on benchmarks.

Auxiliary-Loss-Free Balancing (DeepSeek V3)

Instead of a loss, add bias terms $b_i$ to router logits:

$g_i = \text{softmax}(\text{logit}_i + b_i)$

The biases are not trained by gradient descent. Instead, a simple control rule:

• If expert $i$ is overloaded: decrease $b_i$
• If expert $i$ is underloaded: increase $b_i$

This is a control system operating alongside training, with zero gradient interference. The result: perfect load balance with no quality compromise.

Fine-Grained Experts: 256 with Top-8

Why Not 8 Experts with Top-2?

Mixtral uses 8 experts with top-2 routing: $\binom{8}{2} = 28$ possible combinations per token. DeepSeek V3 uses 256 experts with top-8: $\binom{256}{8} \approx 4.4 \times 10^{13}$ combinations.

More combinations enables finer-grained specialization. Each expert can focus on a narrow knowledge domain (specific language patterns, code syntax, mathematical reasoning) because the combinatorial space is rich enough that tokens always find a relevant subset.

Shared Experts

One expert processes every token regardless of routing. This shared expert captures universal knowledge — function words, basic syntax, common patterns — preventing the routed experts from wasting capacity on things every token needs.

def moe_forward(hidden, shared_expert, routed_experts, router, k=8):
    # Shared expert: processes every token
    shared_output = shared_expert(hidden)

    # Routing
    expert_ids, gates = router(hidden, k=k)

    # Routed experts: only selected ones fire
    routed_output = torch.zeros_like(hidden)
    for i in range(k):
        expert_id = expert_ids[:, i]
        gate = gates[:, i].unsqueeze(-1)
        expert_out = routed_experts[expert_id](hidden)
        routed_output += gate * expert_out

    return shared_output + routed_output

Expert Parallelism: All-to-All Communication

With 256 experts across 64 GPUs (~4 experts per GPU), tokens must be dispatched to the GPU holding their selected experts, then combined back:

Dispatch Phase

1. Router determines which experts each token needs
2. Token hidden states are sent to the GPUs holding those experts
3. This is an all-to-all communication pattern — every GPU potentially sends to every other GPU

Combine Phase

1. Each expert processes its received tokens
2. Expert outputs are sent back to the original GPUs
3. Gated outputs are aggregated

Communication Volume

For batch size $B$, hidden dim $D$, top-$k$ routing, and FP16:

$\text{Dispatch bytes} = B \times k \times D \times 2 \quad \text{(each token sent to k experts)}$

For $B = 4096$, $k = 8$, $D = 7168$ (DeepSeek V3): $\text{Dispatch} = 4096 \times 8 \times 7168 \times 2 = 449 \text{ MB}$ per layer.

DeepEP: Optimized MoE Communication

DeepSeek’s DeepEP library provides two kernel types:

High-throughput kernels: Exploit asymmetric NVLink/RDMA bandwidth. NVLink intra-node: ~160 GB/s. RDMA inter-node: ~50 GB/s. The kernels schedule NVLink and RDMA transfers simultaneously, achieving near-peak utilization on both.

Low-latency kernels: Use pure RDMA without consuming any streaming multiprocessors. This is critical: if all SMs are busy with expert computation, communication kernels can’t launch. By using RDMA-only communication, the SMs remain free for compute, enabling true computation-communication overlap.

Expert Offloading: MoE on Consumer Hardware

When 256 experts don’t fit in GPU memory (e.g., serving on a single GPU), expert offloading moves inactive experts to CPU or SSD:

1. Prediction: The router determines which experts are needed 1-2 layers ahead
2. Prefetch: Predicted experts are loaded from CPU/SSD to GPU asynchronously
3. Compute: Active experts process tokens while next experts are loading
4. Eviction: Used experts are evicted to make room

This enables serving Mixtral 8x7B on a single 24GB GPU (only 2 experts active at once = ~14B parameters in GPU memory).

MoE vs Dense at Different Scales

At small scale (activated params under 7B): dense often wins. Routing overhead (all-to-all communication, gating computation) dominates. Expert specialization doesn’t emerge with too few experts.

At medium scale (7-70B activated): MoE starts winning on quality-per-FLOP. Mixtral 8x7B (13B activated) approaches 70B dense quality.

At large scale (37B+ activated): MoE is the only practical path to frontier quality. DeepSeek V3 demonstrates that 671B MoE matches 405B dense at 1/10 the training cost.

Serving Challenges

MoE models present unique serving difficulties:

1. Memory: All 256 experts must be accessible, even though only 8 fire per token. Total parameter footprint is 671B regardless of activation sparsity.

2. Latency: All-to-all communication adds latency per layer. For inter-node RDMA: ~77us per dispatch+combine = ~1.5ms per layer = ~120ms per forward pass across 80 layers. This is significant for latency-sensitive applications.

3. Load imbalance at serving time: Unlike training where batches are large, serving requests arrive unpredictably. Some experts may be overloaded while others idle. Dynamic batch sizes exacerbate this.

4. Expert parallelism overhead: Compared to tensor parallelism (2 all-reduces per layer at ~10us each on NVLink), expert parallelism requires all-to-all communication that’s harder to overlap.

When to Choose MoE vs Dense

The trend is clear: as models grow, MoE becomes increasingly dominant. Every frontier model in 2025 uses MoE (DeepSeek V3, Mixtral, Grok, Gemini). Dense models (Llama 3) persist primarily because Meta can afford the training compute.