MiniMax-01: Lightning Attention, 4M Token Context, and the Linear Attention Revival

Standard scaled dot-product attention scales as $O(n^2)$ in sequence length. At 1 million tokens, that is $10^{12}$ operations per layer — even with FlashAttention’s IO optimization, this requires enormous compute. MiniMax-01 takes a fundamentally different approach: Lightning Attention, a linear attention mechanism that scales as $O(n)$, combined with a 456B-parameter MoE architecture. The result: training on 1 million tokens and inference extrapolation to 4 million — with a 20-32x longer context window than comparable models.

The Long-Context Problem

Attention’s $O(n^2)$ cost creates an escalating wall:

FlashAttention solves the memory problem (no $O(n^2)$ HBM traffic) but not the compute problem. At 1M tokens, the quadratic FLOPs are simply too expensive for production serving. You need a subquadratic attention mechanism.

Lightning Attention: Linear Scaling

Lightning Attention is MiniMax’s solution — a linear attention variant that processes sequences in $O(n)$ time and memory.

The Linear Attention Idea

Standard attention: $\text{Attn}(Q, K, V) = \text{softmax}(QK^T / \sqrt{d}) \cdot V$

The softmax creates the $O(n^2)$ bottleneck — it requires materializing the full $n \times n$ score matrix. Linear attention removes the softmax and uses a kernel trick:

$\text{LinearAttn}(Q, K, V) = \phi(Q) \cdot (\phi(K)^T V)$

where $\phi$ is a feature map. The key: by computing $\phi(K)^T V$ first (an $O(nd^2)$ operation), then multiplying by $\phi(Q)$ ($O(nd)$), we avoid the $O(n^2)$ matrix. Total cost: $O(nd^2)$ — linear in $n$.

Why Previous Linear Attention Failed

Performer (2020), Katformer, and other linear attention variants achieved the $O(n)$ scaling but with significant quality degradation. The softmax in standard attention performs two critical functions:

1. Normalization: Attention weights sum to 1, creating a proper weighted average
2. Sharpening: The exponential amplifies score differences, allowing focused attention on relevant tokens

Without softmax, attention distributions become too uniform — the model can’t focus. Quality drops by 2-5 perplexity points, making linear attention impractical for frontier models.

Lightning Attention’s Innovation

Lightning Attention addresses both problems through a hybrid approach:

1. Improved kernel function: Instead of naive ReLU or ELU feature maps, Lightning Attention uses a carefully designed $\phi$ that preserves the sharpening property of softmax while remaining computationally efficient.

2. Chunk-wise computation: Sequences are divided into chunks. Within each chunk, the attention can use a more precise local computation. Across chunks, the linear formulation carries information forward through a compressed state.

3. Integration with the MoE FFN: The linear attention mechanism is co-designed with the MoE layers. Experts can specialize for different regions of long contexts — some experts handle local patterns (recent tokens), others handle global patterns (distant context).

Architecture: 456B MoE with Lightning Attention

The critical differentiator: Lightning Attention’s linear compute scaling. While DeepSeek V3 and Kimi K2 reduce KV cache memory through MLA compression, they still pay $O(n^2)$ compute for attention. MiniMax-01 pays $O(n)$ for both compute and memory.

Training for 1M Context

Training on 1M-token sequences with 456B parameters requires solving several problems:

Memory Management

A 1M-token sequence at d_model=8192 requires massive activation memory. Solutions:

• Activation checkpointing: Recompute activations during backward pass instead of storing them
• Sequence parallelism: Distribute the sequence across multiple GPUs, each holding a segment
• Progressive context extension: Train initially on shorter sequences (32K), gradually extend to 128K, 512K, then 1M

Computation-Communication Overlap

With sequence distributed across GPUs, Lightning Attention’s linear formulation enables efficient distributed computation. Each GPU processes its chunk and passes a compressed state to the next — no all-to-all communication needed for attention (unlike Ring Attention, which must pass KV blocks in a ring).

Lightning Attention’s compressed state is only $O(d^2)$ per layer — vastly smaller than Ring Attention’s $O(nd)$ KV blocks. This is why MiniMax can train at 1M tokens efficiently.

Extrapolation to 4M Tokens

MiniMax-01 extrapolates from 1M training context to 4M inference context. This works because:

1. Linear attention has no position-dependent components that break: Unlike RoPE, where unseen rotation angles cause attention score degradation, Lightning Attention’s state-based formulation naturally handles longer sequences.

2. The compressed state carries sufficient information: The $O(d^2)$ state matrix accumulates context information across the entire sequence. As long as the state has capacity to represent the relevant information, additional tokens can be processed.

3. MoE routing adapts: Different experts activate for different parts of the context, effectively increasing the model’s capacity for longer inputs without proportional compute increase.

Performance Analysis

MiniMax-01’s strength is clear: it matches GPT-4o and Claude on standard benchmarks while offering 8-20x longer context windows. The tradeoff: slightly lower scores on code generation (HumanEval), likely because the linear attention mechanism loses some of the precise token-level focus that softmax attention provides for code.

Implications for Serving

4M-token context creates new serving challenges:

• Memory: KV cache for 4M tokens at d=8192 with 32 heads: $4M \times 32 \times 128 \times 2 \times 2 = 64$ GB per layer. Across 64 layers: 4 TB. This doesn’t fit on any single GPU.
• Throughput: Processing a 4M-token prompt takes minutes even with linear attention. Batch size is effectively 1 for very long contexts.
• Serving pattern: Long-context requests are rare but expensive. A disaggregated architecture (covered in Inference Timeline Part 10) helps: dedicate specific nodes to long-context prefill while others handle short-context decode.

What MiniMax-01 Means for the Field

MiniMax-01 demonstrates that linear attention is viable for frontier-quality models when:

1. The kernel function is carefully designed (not naive ReLU/ELU)
2. The architecture is co-optimized (MoE + Lightning Attention)
3. Training is progressive (short to long context)
4. The use case genuinely requires very long context (1M+ tokens)

For most applications, standard attention + FlashAttention + RoPE scaling remains the pragmatic choice. But for document-scale processing, code repository understanding, and multi-document reasoning, Lightning Attention opens possibilities that quadratic attention simply cannot reach.

The frontier model landscape in 2025 now has two viable attention paradigms: softmax-based (DeepSeek V3, Kimi K2, Llama 4) and linear (MiniMax-01, with Mamba hybrids as a third path). The next post in this series surveys where all frontier models are converging and where they diverge.