Reasoning Scaling Laws: How Inference-Time Compute Changes Everything We Know About Scaling

For a decade, the LLM scaling playbook was simple: more training compute yields better models. Chinchilla (2022) formalized this into a law: $L(N, D) \propto N^{-\alpha} + D^{-\beta}$ where $N$ is parameters and $D$ is training tokens. Invest in training, reap quality at inference.

In 2024-2025, a second scaling axis emerged: inference-time compute. Models like o1, DeepSeek-R1, and QwQ show that generating more “thinking” tokens at inference improves answer quality — sometimes by 30%+ on reasoning benchmarks. This changes the economics of AI fundamentally: you can now trade inference cost for quality, not just training cost.

The Two-Axis Scaling Framework

The critical insight: $\alpha$ and $\gamma$ are independent scaling exponents. You can improve quality by training more OR by thinking more. For a fixed total compute budget (training + inference amortized over queries), there is an optimal split.

The curve is logarithmic: the first 500 thinking tokens provide the largest quality jump. Each subsequent doubling adds less. This shapes the compute-optimal allocation strategy.

How Reasoning Models Think

Standard LLMs generate answers directly: prompt in, answer out. Reasoning models insert an intermediate “thinking” phase:

Standard:  prompt -> [model] -> answer
Reasoning: prompt -> [model] -> thinking_tokens -> answer

The thinking tokens are a chain-of-thought (CoT) — the model’s internal working. For a math problem: “First, let me identify the variables. We have x = 3 and y = 5. The equation is x^2 + y^2 = z. So z = 9 + 25 = 34.”

Why does this help? Two complementary theories:

1. Effective circuit depth: A transformer has fixed depth (e.g., 80 layers). Complex reasoning may require more sequential computation steps than 80 layers can provide. Thinking tokens allow the model to “unroll” its computation across multiple forward passes — each thinking token adds 80 more layers of processing.

2. Working memory: The model’s hidden state is fixed-size ($d_{\text{model}}$ dimensions). Thinking tokens externalize intermediate results into the token sequence (effectively the KV cache), providing unbounded working memory.

DeepSeek-R1: Training a Reasoning Model

DeepSeek-R1 pioneered open-source reasoning models. The training recipe has three stages:

Stage 1: Supervised Fine-Tuning (SFT)

Start from a strong base model (DeepSeek-V3). Fine-tune on high-quality reasoning traces: (problem, chain-of-thought, answer) triples. The CoT traces are generated by the model itself or by stronger models.

Stage 2: Reinforcement Learning with GRPO

Group Relative Policy Optimization (GRPO) is DeepSeek’s alternative to PPO (Proximal Policy Optimization). The key difference: no critic network.

For each problem, generate $K$ candidate solutions. Compute a reward for each (correctness, format compliance). Then optimize the policy relative to the group:

def grpo_loss(model, problems, K=8, clip_eps=0.2, kl_coeff=0.01):
    """Group Relative Policy Optimization loss."""
    total_loss = 0.0

    for problem in problems:
        # Generate K completions
        completions = [model.generate(problem) for _ in range(K)]
        rewards = [reward_fn(problem, c) for c in completions]

        # Compute group-relative advantages
        mean_reward = sum(rewards) / K
        std_reward = (sum((r - mean_reward)**2 for r in rewards) / K) ** 0.5
        advantages = [(r - mean_reward) / (std_reward + 1e-8) for r in rewards]

        # Policy gradient with clipping
        for completion, advantage in zip(completions, advantages):
            log_prob = model.log_prob(completion)
            old_log_prob = log_prob.detach()  # No separate old policy needed
            ratio = (log_prob - old_log_prob).exp()
            clipped = ratio.clamp(1 - clip_eps, 1 + clip_eps)
            pg_loss = -min(ratio * advantage, clipped * advantage)
            kl_penalty = kl_coeff * (log_prob - old_log_prob)
            total_loss += pg_loss + kl_penalty

    return total_loss / len(problems)

Stage 3: Rejection Sampling + SFT

After RL, generate many solutions per problem. Keep only those that are both correct AND use clean reasoning (no gibberish steps, proper formatting). Fine-tune on this filtered set. This “distills” the RL policy into a cleaner SFT model.

Compute-Optimal Inference Allocation

For a given query, how many thinking tokens are optimal?

Define: difficulty $d$ (estimated from the query), quality gained per thinking token $g(T, d)$, cost per token $c$. The optimal thinking budget:

$T^*(d) = \arg\max_T \left[ V \cdot Q(T, d) - T \cdot c \right]$

where $V$ is the value of a correct answer and $Q(T, d)$ is the probability of correctness at $T$ thinking tokens for difficulty $d$.

With the logarithmic quality model $Q(T, d) \approx Q_0(d) + \gamma(d) \ln(T)$:

$T^*(d) = \frac{V \cdot \gamma(d)}{c}$

Adaptive Thinking Budgets

Production systems cannot afford to let every query think for 50,000 tokens. The model must decide how much to think:

def adaptive_generate(model, prompt, max_think_tokens=10000, confidence_threshold=0.95):
    """Generate with adaptive thinking budget."""
    tokens = tokenize(prompt)
    thinking_tokens = []

    for step in range(max_think_tokens):
        next_token = model.generate_one(tokens + thinking_tokens)

        # Check if model signals it is done thinking
        if next_token == ANSWER_TOKEN:
            break

        # Check confidence: if model is very sure, stop early
        logit_entropy = compute_entropy(model.last_logits)
        if logit_entropy < confidence_threshold and step > 100:
            break  # Model is confident enough

        thinking_tokens.append(next_token)

    # Generate final answer
    answer = model.generate(tokens + thinking_tokens + [ANSWER_TOKEN])
    return answer, len(thinking_tokens)

Systems Implications

Reasoning models change every aspect of serving infrastructure.

KV Cache Explosion

A standard query generates 100-500 output tokens. A reasoning query generates 1,000-50,000 thinking tokens PLUS the answer. The KV cache grows proportionally:

At 50K thinking tokens, a single query consumes 16.8 GB of KV cache — a single H100 can only serve 3 such queries concurrently. This is a fundamental shift from serving 130 standard queries.

Attention Cost

Attention is $O(n^2)$. At 50K tokens total context: $50000^2 = 2.5$ billion operations per layer. Even with FlashAttention, this is 50x more compute than a standard query. Reasoning queries are both memory-intensive (KV cache) and compute-intensive (quadratic attention).

Scheduling Challenges

Standard queries have predictable decode length (100-500 tokens). Reasoning queries vary from 200 to 50,000 tokens — a 250x range. This makes batch scheduling much harder:

• Resource prediction: The scheduler cannot predict how much KV cache a reasoning query will need
• Preemption risk: A reasoning query that unexpectedly generates 50K tokens may force preemption of other queries
• Batch heterogeneity: Mixing standard and reasoning queries in one batch leads to poor utilization (the short queries finish and their slots sit empty while the long query continues)

Cost Impact

A 50K-token reasoning query costs 100x more than a standard query. This is why reasoning models are typically reserved for high-value tasks (code generation, math, complex analysis) where the quality improvement justifies the cost.

Verification: Checking Reasoning Quality

How do you know if the model’s thinking led to a correct answer?

Majority Voting

Generate $N$ independent reasoning traces. Take the most common answer. If 6 of 8 traces produce “34”, that’s the answer.

Success probability with $N$ samples and per-sample accuracy $p$:

$P_{\text{majority}}(N, p) = \sum_{k=\lceil N/2 \rceil}^{N} \binom{N}{k} p^k (1-p)^{N-k}$

For $p = 0.7$ and $N = 8$: $P_{\text{majority}} \approx 0.94$. Much better than $p = 0.7$ alone.

Process Reward Models (PRMs)

Score each step in the reasoning chain, not just the final answer. A PRM assigns a score to each thinking step. The product of step scores gives the overall chain quality:

$\text{PRM\_score}(chain) = \prod_{i=1}^{S} \text{step\_score}(s_i)$

Select the chain with the highest PRM score. This is more effective than majority voting because it evaluates reasoning quality, not just answer consensus.

What This Means for the Field

The reasoning scaling law introduces a new degree of freedom in AI system design:

1. Smaller models + more thinking can match larger models on reasoning tasks. A 7B model with 10K thinking tokens can outperform a 70B model with direct generation on some math/code tasks.

2. Inference cost becomes the dominant expense for reasoning-heavy workloads. Training cost is amortized over billions of queries, but inference cost is paid per query.

3. Serving infrastructure must adapt: variable-length generation, KV cache for 50K+ token sequences, scheduling that handles 250x variance in query cost.

4. New optimization targets: instead of minimizing per-token latency, minimize cost-per-correct-answer. Sometimes generating 8 cheap samples and voting is cheaper than generating 1 expensive long chain.

The inference-time scaling law is not a replacement for training-time scaling — it is a complement. The future of AI scaling is two-dimensional: train strong base models AND think harder at inference. The optimal point on this frontier depends on the specific task, cost constraints, and latency requirements.