DeepSeek V3: How 671B Parameters Trained for the Cost of a 70B Dense Model

DeepSeek V3 is a 671B-parameter Mixture-of-Experts model that activates only 37B parameters per token. It was pre-trained on 14.8 trillion tokens using just 2.788 million H800 GPU hours — a fraction of what comparably-performing models cost. It outperforms Llama 3 405B on most benchmarks while using roughly 10x less training compute.

This post analyzes the five architectural innovations that make this efficiency possible: Multi-head Latent Attention (MLA), fine-grained MoE, auxiliary-loss-free load balancing, FP8 training, and DualPipe scheduling.

Multi-head Latent Attention (MLA)

The Problem MLA Solves

Standard GQA (used by Llama 3) stores separate K and V tensors for each KV group in the cache. For Llama 3 70B with 8 KV heads and $d_h = 128$:

$\text{KV per token per layer} = 2 \times 8 \times 128 \times 2 = 4{,}096 \text{ bytes (FP16)}$

Across 80 layers and a 4K context: $4{,}096 \times 80 \times 4{,}096 = 1.28 \text{ GB per sequence}$. At batch=64, that’s 82 GB — more than the model weights.

The MLA Insight

Instead of caching full K and V heads, MLA compresses them into a single low-rank latent vector $c_t^{KV}$:

$c_t^{KV} = W^{DKV} h_t \quad \text{where } c_t^{KV} \in \mathbb{R}^{d_c}, \; d_c \ll n_h \times d_h$

During attention, K and V are reconstructed on-the-fly:

$K_t = W^{UK} c_t^{KV}, \quad V_t = W^{UV} c_t^{KV}$

The cache stores only $c_t^{KV}$ — a vector of dimension $d_c = 512$ instead of $n_h \times d_h = 8 \times 128 = 1{,}024$ for each of K and V (2,048 total). That’s a 75% reduction just from the latent compression.

The Absorption Trick

The real magic: during inference, the up-projection $W^{UK}$ can be absorbed into the query projection. Instead of:

$\text{score} = q_t^T (W^{UK} c_s^{KV})$

We precompute $\hat{W}^Q = W^Q \times W^{UK}$ and compute:

$\text{score} = (\hat{W}^Q h_t)^T c_s^{KV}$

This means we never materialize the full K tensor at all. The same trick applies to V with the output projection. Result: 93.3% KV cache reduction vs standard MHA with zero quality loss.

Fine-Grained MoE with 256 Experts

Why More Experts

Switch Transformer used 128 experts with top-1 routing. Mixtral uses 8 experts with top-2. DeepSeek V3 uses 256 routed experts with top-8 routing, plus 1 shared expert.

The reasoning is combinatorial: with top-8 selection from 256 experts, each token can activate $\binom{256}{8} \approx 4.4 \times 10^{13}$ unique expert combinations. With top-2 from 8, only $\binom{8}{2} = 28$ combinations exist. More combinations means finer-grained specialization — each expert can focus on a narrower knowledge domain.

Shared Experts

One expert processes every token regardless of routing. This shared expert captures common knowledge (function words, basic syntax, universal patterns) that every token needs, preventing the routed experts from wasting capacity on common patterns.

The Math

Each token’s computation:

$h = h + \text{SharedExpert}(h) + \sum_{i \in \text{TopK}(g(h), 8)} g_i(h) \cdot \text{Expert}_i(h)$

where $g(h)$ is the gating function producing router logits. Total activated parameters per token: 1 shared expert + 8 routed experts = 9 expert FFNs + attention ≈ 37B of the 671B total.

Auxiliary-Loss-Free Load Balancing

The Problem with Auxiliary Losses

Standard MoE training adds a load-balancing auxiliary loss that penalizes uneven expert utilization. The problem: this loss competes with the language modeling objective, distorting gradients and reducing model quality.

DeepSeek’s Solution: Bias Terms

Instead of a gradient-based loss, DeepSeek V3 adds learnable bias terms $b_i$ to router logits:

$g_i = \text{softmax}(\text{logit}_i + b_i)$

These biases are NOT updated through backpropagation. Instead, a simple rule: if expert $i$ is overloaded, decrease $b_i$; if underloaded, increase $b_i$. This is essentially a control system (PID-like) operating alongside gradient descent, with zero interference to the training signal.

Multi-Token Prediction (MTP)

Standard LLM training predicts only the next token. DeepSeek V3 trains additional prediction heads that predict tokens 2, 3, …, K steps ahead simultaneously.

Training Benefits

1. Richer gradient signal: Each position receives feedback from multiple future tokens, not just the immediate next one
2. Better representation learning: The model must maintain representations useful for multi-step prediction, leading to more robust features

Inference Benefits: Self-Speculation

The MTP heads serve as a built-in draft model for speculative decoding — no separate draft model needed. The main model generates K candidate tokens in parallel, then verifies them in one forward pass. Expected speedup: 1.8x at batch=1 with acceptance rate ~0.85.

FP8 Mixed-Precision Training

DeepSeek V3 trained the entire 671B model in FP8 — the first model at this scale to do so without loss spikes or rollbacks.

Format Selection

• E4M3 (4 exponent, 3 mantissa bits): Used for forward pass GEMMs. More precision for activations and weights.
• E5M2 (5 exponent, 2 mantissa bits): Used for backward pass. More dynamic range for gradients.
• FP32: Used for master weights, optimizer states, LayerNorm, softmax, routing — anything sensitive to numerical precision.

Per-Tensor Delayed Scaling

Each tensor (weights, activations, gradients) gets its own dynamic scale factor. “Delayed” means the scale is computed from the previous iteration’s statistics, avoiding an extra pass over the data:

# Delayed scaling: use previous iteration's amax
scale = (448.0 / amax_history[-1]).clamp(max=max_scale)
x_fp8 = (x * scale).to(torch.float8_e4m3fn)
amax_history.append(x.abs().max())  # Record for next iteration

Result

FP8 training achieves ~1.8x throughput over BF16 on H800 GPUs, while maintaining training stability across the full 14.8T token run. The 2.788M GPU-hour budget would have been ~5M GPU-hours in BF16.

DualPipe: Near-Zero Pipeline Bubbles

Standard Pipeline Parallelism

With $P$ pipeline stages and $M$ micro-batches, 1F1B scheduling has a bubble fraction of $(P-1)/(P-1+M)$. For $P=16$ and $M=64$: 19% of GPU time is idle. At DeepSeek’s scale, this translates to millions of wasted GPU-hours.

DualPipe

DualPipe sends micro-batches from both ends of the pipeline simultaneously:

• Forward micro-batches flow left-to-right (standard)
• Additional micro-batches flow right-to-left simultaneously
• Forward and backward passes of different micro-batches overlap on each GPU

The result: pipeline bubbles approach zero because each GPU always has work from one direction or the other.

Training Infrastructure

DeepSeek V3 was trained on 2,048 H800 GPUs across 256 nodes with a 5D parallelism strategy:

Communication is optimized with DeepEP — a custom all-to-all library for MoE dispatch/combine that exploits asymmetric NVLink (160 GB/s intra-node) and RDMA (50 GB/s inter-node) bandwidth.

What’s Portable from DeepSeek V3

Not every idea requires 2,048 GPUs. Several innovations are immediately applicable: