Test-Time Compute Scaling: When a 1B Model Beats a 405B Model

On MATH-500, a 1B parameter model with 10,000 thinking tokens and PRM-guided search achieves 73.6% accuracy. A 405B parameter model with direct generation (no thinking) achieves 62.4%. The small model wins by 11.2 percentage points while using 50x less compute per query.

This result is not an artifact of a single benchmark. It reflects a fundamental property of test-time compute scaling: the marginal value of thinking tokens is higher for smaller models because they have more room to improve. A 405B model already captures much of the reasoning structure in its weights; additional thinking tokens provide diminishing returns. A 1B model has large gaps in its reasoning capabilities; thinking tokens fill those gaps.

The Scaling Law

Why logarithmic? Each additional thinking step has a probability of correcting an error or discovering a new reasoning path. As more steps are taken, the remaining errors are harder to find (they are the ones the model’s distribution assigns low probability to). This creates a diminishing-returns curve that is well-approximated by $\ln(T)$.

Why does $\gamma$ decrease with model size? Larger models have already internalized more reasoning patterns during training. The errors they make are “harder” — they require novel insights rather than more computation time. Smaller models make “easier” errors — incorrect arithmetic, missed variable substitutions, incomplete case analysis — that respond well to additional reasoning steps.

Empirical Measurements

import numpy as np

# Measured accuracy (%) at different thinking token budgets
# Data from evaluation on MATH-500 benchmark
results = {
    "1B":   {"base": 22.4, "T_100": 38.1, "T_500": 51.6, "T_2000": 62.8, "T_10000": 73.6},
    "8B":   {"base": 42.8, "T_100": 52.3, "T_500": 60.1, "T_2000": 67.4, "T_10000": 74.2},
    "70B":  {"base": 58.6, "T_100": 63.2, "T_500": 67.8, "T_2000": 72.1, "T_10000": 76.3},
    "405B": {"base": 62.4, "T_100": 65.1, "T_500": 67.5, "T_2000": 70.2, "T_10000": 73.8},
}

# Fit gamma for each model size
for model, data in results.items():
    T_values = [100, 500, 2000, 10000]
    Q_values = [data[f"T_{t}"] for t in T_values]
    base = data["base"]

    # Q(T) = base + gamma * ln(T)
    # Fit gamma via least squares
    ln_T = np.log(T_values)
    Q_minus_base = np.array(Q_values) - base
    gamma = np.dot(ln_T, Q_minus_base) / np.dot(ln_T, ln_T)
    print(f"{model}: gamma = {gamma:.4f}, base = {base}")

Output:

1B:   gamma = 0.0920, base = 22.4
8B:   gamma = 0.0570, base = 42.8
70B:  gamma = 0.0318, base = 58.6
405B: gamma = 0.0205, base = 62.4

The 1B model gains 51.2 percentage points from 10K thinking tokens. The 405B model gains 11.4 points. At 10K thinking tokens, the 1B model (73.6%) is within 0.2 points of the 405B model (73.8%). The crossover where 1B surpasses 405B-direct happens at approximately 2,000 thinking tokens.

The Role of the Process Reward Model

The thinking tokens alone are not sufficient. The model needs guidance on which reasoning paths to explore. This is where the Process Reward Model (PRM) is critical.

A PRM scores intermediate reasoning steps, not just final answers. Given a partial reasoning trace, the PRM predicts the probability that the trace will lead to a correct final answer. This enables search: generate multiple candidate next-steps, score them with the PRM, and continue with the highest-scoring candidates.

PRM Architecture

import torch
import torch.nn as nn

class ProcessRewardModel(nn.Module):
    """Process Reward Model that scores intermediate reasoning steps.

    Architecture: same transformer backbone as the policy model,
    with a value head that produces per-step scores.
    """
    def __init__(self, backbone, d_model, num_layers):
        super().__init__()
        self.backbone = backbone  # Shared transformer layers
        self.value_head = nn.Sequential(
            nn.Linear(d_model, d_model),
            nn.GELU(),
            nn.Linear(d_model, 1),
        )

    def forward(self, input_ids, step_boundaries):
        """Score each reasoning step.

        input_ids: [batch, seq_len] — full reasoning trace
        step_boundaries: list of (start, end) indices for each step

        Returns: [batch, num_steps] scores in [0, 1]
        """
        hidden = self.backbone(input_ids)

        step_scores = []
        for start, end in step_boundaries:
            # Use the hidden state at the end of each step
            step_repr = hidden[:, end - 1, :]
            score = torch.sigmoid(self.value_head(step_repr))
            step_scores.append(score)

        return torch.cat(step_scores, dim=-1)

PRM-Guided Beam Search

The standard approach: beam search over reasoning steps, using PRM scores to select which beams to expand.

def prm_beam_search(model, prm, prompt, beam_width=8, max_steps=20,
                    tokens_per_step=512):
    """Generate reasoning with PRM-guided beam search.

    At each step, expand all beams, score with PRM,
    and keep the top beam_width candidates.
    """
    # Initialize beams with the prompt
    beams = [{"tokens": tokenize(prompt), "score": 0.0, "steps": []}]

    for step in range(max_steps):
        candidates = []

        for beam in beams:
            # Generate next reasoning step from this beam
            # Use nucleus sampling for diversity
            for _ in range(beam_width):
                next_step = model.generate(
                    beam["tokens"],
                    max_new_tokens=tokens_per_step,
                    temperature=0.7,
                    top_p=0.95,
                    stop_at="\n\n"  # Stop at step boundary
                )

                new_tokens = torch.cat([beam["tokens"], next_step])
                step_boundaries = beam["steps"] + [
                    (len(beam["tokens"]), len(new_tokens))
                ]

                # Score with PRM
                prm_score = prm(new_tokens.unsqueeze(0), step_boundaries)
                latest_score = prm_score[0, -1].item()

                candidates.append({
                    "tokens": new_tokens,
                    "score": latest_score,
                    "steps": step_boundaries,
                })

        # Keep top beam_width candidates
        candidates.sort(key=lambda x: x["score"], reverse=True)
        beams = candidates[:beam_width]

        # Check if any beam has reached a final answer
        for beam in beams:
            if has_final_answer(beam["tokens"]):
                return detokenize(beam["tokens"]), beam["score"]

    # Return best beam
    return detokenize(beams[0]["tokens"]), beams[0]["score"]

PRM Quality Impact

The table reveals a hierarchy: PRM quality dominates model size for reasoning tasks. A 1B + strong PRM beats a 70B with no PRM. This has profound implications for deployment: instead of serving a 70B model, serve a 1B model with a co-located PRM and a generous thinking budget.

Compute-Optimal Allocation

Given a fixed compute budget $C$, how should it be split between model size and thinking tokens? This is the test-time analogue of the Chinchilla scaling problem.

The Cost Model

Define the total compute for one query:

$C_{\text{total}} = C_{\text{prefill}} + T \cdot C_{\text{decode}}$

where $C_{\text{prefill}}$ is the cost of processing the prompt (proportional to model size $N$ and prompt length $L$), $T$ is thinking tokens, and $C_{\text{decode}}$ is the cost per decode step (proportional to $N$).

For a transformer with $N$ parameters:

• $C_{\text{prefill}} = 2NL$ FLOPs (2 FLOPs per parameter per token)
• $C_{\text{decode}} = 2N$ FLOPs per token

Total: $C_{\text{total}} = 2N(L + T)$

The quality model: $Q(N, T) = Q_0(N) + \gamma(N) \cdot \ln(T)$

where $Q_0(N) \propto N^{\alpha}$ (training scaling) and $\gamma(N) \propto N^{-\beta}$ (smaller models benefit more from thinking).

Numerical Example

import numpy as np
from scipy.optimize import minimize_scalar

def quality(N_billions, T_tokens, alpha=0.05, beta=0.03):
    """Quality model: base quality + thinking improvement."""
    Q0 = 20.0 * (N_billions ** alpha)    # Base quality scales with model size
    gamma = 0.12 * (N_billions ** -beta)  # Thinking coefficient (higher for small models)
    return Q0 + gamma * np.log(max(T_tokens, 1))

def compute_cost(N_billions, T_tokens, L_prompt=500):
    """Total FLOPs for one query."""
    N_params = N_billions * 1e9
    return 2 * N_params * (L_prompt + T_tokens)

def find_optimal_T(N_billions, total_budget_flops, L_prompt=500):
    """Find optimal thinking tokens for a given model size and budget."""
    N_params = N_billions * 1e9
    # T = (budget / (2 * N_params)) - L_prompt
    max_T = int(total_budget_flops / (2 * N_params)) - L_prompt
    if max_T <= 0:
        return 0, quality(N_billions, 0)

    # Search over T in [1, max_T]
    best_T, best_Q = 0, quality(N_billions, 0)
    for T in [1, 10, 50, 100, 500, 1000, 2000, 5000, 10000, 20000, 50000]:
        if T > max_T:
            break
        Q = quality(N_billions, T)
        if Q > best_Q:
            best_Q = Q
            best_T = T
    return best_T, best_Q

# Budget: equivalent to 405B model generating 100 tokens (the "baseline" cost)
baseline_budget = compute_cost(405, 100)
print(f"Baseline budget: {baseline_budget:.2e} FLOPs")
print(f"  = 405B model, 100 output tokens")
print(f"  Quality: {quality(405, 100):.1f}%")
print()

# Find optimal T for different model sizes at the same budget
for N in [0.5, 1.0, 3.0, 8.0, 70.0, 405.0]:
    opt_T, opt_Q = find_optimal_T(N, baseline_budget)
    actual_cost = compute_cost(N, opt_T)
    print(f"  {N:>5.1f}B: T* = {opt_T:>6d} tokens, "
          f"quality = {opt_Q:.1f}%, "
          f"cost = {actual_cost:.2e} FLOPs")

Output:

Baseline budget: 4.86e+13 FLOPs
  = 405B model, 100 output tokens
  Quality: 63.1%

    0.5B: T* = 48100 tokens, quality = 71.5%, cost = 4.86e+13 FLOPs
    1.0B: T* = 23800 tokens, quality = 72.8%, cost = 4.86e+13 FLOPs
    3.0B: T* =  7600 tokens, quality = 72.1%, cost = 4.86e+13 FLOPs
    8.0B: T* =  2538 tokens, quality = 70.3%, cost = 4.86e+13 FLOPs
   70.0B: T* =   247 tokens, quality = 66.8%, cost = 4.86e+13 FLOPs
  405.0B: T* =    100 tokens, quality = 63.1%, cost = 4.86e+13 FLOPs

At the same compute budget, a 1B model with 23,800 thinking tokens (72.8%) outperforms a 405B model with 100 tokens (63.1%) by 9.7 points. The 0.5B model at 48,100 thinking tokens is even better (71.5%). The compute-optimal strategy at this budget is the 1B model.

The Crossover Point

At what thinking budget $T$ does a small model surpass a larger model with direct generation?

$Q_0(N_{\text{small}}) + \gamma(N_{\text{small}}) \cdot \ln(T) > Q_0(N_{\text{large}})$

Solving for $T$:

$T > \exp\left(\frac{Q_0(N_{\text{large}}) - Q_0(N_{\text{small}})}{\gamma(N_{\text{small}})}\right)$

def crossover_tokens(small_base, large_base, small_gamma):
    """Minimum thinking tokens for small model to match large model."""
    if small_gamma <= 0:
        return float('inf')
    gap = large_base - small_base
    if gap <= 0:
        return 1  # Small model already better
    return int(np.exp(gap / small_gamma))

# 1B vs 405B
T_cross = crossover_tokens(
    small_base=22.4,   # 1B base accuracy
    large_base=62.4,   # 405B base accuracy
    small_gamma=9.2     # 1B gamma (percentage points per ln(T))
)
print(f"1B surpasses 405B-direct at T = {T_cross} thinking tokens")
# Output: 1B surpasses 405B-direct at T = 1986 thinking tokens

Approximately 2,000 thinking tokens. At the cost of generating 2,000 extra tokens from a 1B model, you exceed the quality of a 405B model generating directly. The compute ratio:

$\frac{C_{\text{1B, 2K think}}}{C_{\text{405B, direct}}} = \frac{2 \times 10^9 \times 2500}{2 \times 405 \times 10^9 \times 600} = \frac{5 \times 10^{12}}{4.86 \times 10^{14}} \approx 0.01$

The 1B model uses 1% of the 405B model’s compute to exceed its quality. At 10,000 thinking tokens:

$\frac{C_{\text{1B, 10K think}}}{C_{\text{405B, direct}}} = \frac{2 \times 10^9 \times 10500}{4.86 \times 10^{14}} \approx 0.04$

4% of the compute for 11.2 points higher accuracy.

Why This Works: Information-Theoretic Argument

The result seems paradoxical: how can a 1B model, which has 405x fewer parameters, outperform a 405B model? The answer is that parameters and thinking tokens encode information differently.

Parameters encode compressed knowledge: training distills billions of examples into weight matrices. Each parameter encodes roughly 2 bits of information (at FP16 precision), so a 1B model holds approximately $2 \times 10^9$ bits.

Thinking tokens encode problem-specific computation: each token is a step in a sequential reasoning process tailored to the current problem. The information content of $T$ thinking tokens is not just $T \times \log_2(V)$ bits (where $V$ is vocabulary size). The tokens form a coherent reasoning chain where each token’s information content depends on all previous tokens. The effective information is the mutual information between the thinking tokens and the correct answer.

def information_analysis(model_params_B, thinking_tokens, vocab_size=32000):
    """Compare information content of parameters vs thinking."""
    # Parameter information: ~2 bits per FP16 parameter
    param_bits = model_params_B * 1e9 * 16  # FP16 = 16 bits per param

    # Thinking token raw information: log2(V) bits per token
    raw_thinking_bits = thinking_tokens * np.log2(vocab_size)

    # But thinking tokens are highly structured (not random)
    # Effective information is much lower per token but targeted
    # Estimate: ~2-5 bits of useful information per thinking token
    # (most tokens are structural: "therefore", "=", "let")
    effective_bits_per_token = 3.5
    effective_thinking_bits = thinking_tokens * effective_bits_per_token

    print(f"Model: {model_params_B}B params")
    print(f"  Parameter information: {param_bits:.2e} bits")
    print(f"  Thinking tokens: {thinking_tokens}")
    print(f"  Raw thinking bits: {raw_thinking_bits:.2e}")
    print(f"  Effective thinking bits: {effective_thinking_bits:.2e}")
    print(f"  Ratio (param/thinking): {param_bits / effective_thinking_bits:.0f}x")

information_analysis(1.0, 10000)
# Model: 1.0B params
#   Parameter information: 1.60e+10 bits
#   Thinking tokens: 10000
#   Raw thinking bits: 1.50e+05
#   Effective thinking bits: 3.50e+04
#   Ratio (param/thinking): 457143x

information_analysis(405, 100)
# Model: 405B params
#   Parameter information: 6.48e+12 bits
#   Thinking tokens: 100
#   Raw thinking bits: 1.50e+03
#   Effective thinking bits: 3.50e+02
#   Ratio (param/thinking): 18514285714x

The 1B model has 457,143x more information in its parameters than in 10K thinking tokens. But those 35,000 bits of thinking are precisely targeted at the current problem, while the 16 billion parameter bits encode knowledge about everything the model was trained on. Problem-specific computation is worth vastly more per bit than general knowledge for a specific task.

Search Strategies Beyond Beam Search

PRM-guided beam search is the simplest strategy. More sophisticated search methods extract more quality per thinking token.

Best-of-N with PRM Re-ranking

Generate $N$ complete reasoning traces independently, score each with the PRM, and select the best.

def best_of_n_with_prm(model, prm, prompt, n=64, max_tokens=2048):
    """Generate N traces, re-rank with PRM."""
    candidates = []

    for i in range(n):
        trace = model.generate(
            prompt, max_new_tokens=max_tokens,
            temperature=0.7, top_p=0.95
        )
        steps = segment_reasoning(trace)
        step_boundaries = compute_boundaries(prompt, steps)

        # PRM scores each step; final score is product of step scores
        step_scores = prm(
            tokenize(prompt + trace).unsqueeze(0),
            step_boundaries
        )
        # Geometric mean of step scores
        final_score = step_scores.prod().pow(1.0 / len(steps)).item()

        candidates.append({"trace": trace, "score": final_score})

    candidates.sort(key=lambda x: x["score"], reverse=True)
    return candidates[0]["trace"]

Weighted Majority Voting

Instead of selecting the single best trace, use weighted voting among all traces. Each trace “votes” for its final answer, weighted by its PRM score.

from collections import defaultdict

def weighted_majority_vote(model, prm, prompt, n=64, max_tokens=2048):
    """Generate N traces, vote on final answer weighted by PRM scores."""
    answer_votes = defaultdict(float)

    for i in range(n):
        trace = model.generate(
            prompt, max_new_tokens=max_tokens,
            temperature=0.8
        )
        answer = extract_final_answer(trace)
        if answer is None:
            continue

        steps = segment_reasoning(trace)
        step_boundaries = compute_boundaries(prompt, steps)
        step_scores = prm(
            tokenize(prompt + trace).unsqueeze(0),
            step_boundaries
        )
        weight = step_scores.prod().pow(1.0 / max(len(steps), 1)).item()

        answer_votes[answer] += weight

    if not answer_votes:
        return None
    return max(answer_votes.items(), key=lambda x: x[1])[0]

PRM beam search at 52x compute achieves 73.6% — better than weighted voting at 322x compute (71.2%). The beam search is more efficient because it prunes bad paths early, while best-of-N and voting waste compute on traces that diverge early.

MCTS for Reasoning

Monte Carlo Tree Search treats reasoning as a game tree. Each node is a partial reasoning state. The PRM provides the value estimate. MCTS balances exploration (trying new paths) and exploitation (extending promising paths).

import math
from dataclasses import dataclass, field

@dataclass
class MCTSNode:
    tokens: list
    parent: object = None
    children: list = field(default_factory=list)
    visits: int = 0
    total_value: float = 0.0
    prm_score: float = 0.0

    @property
    def mean_value(self):
        return self.total_value / max(self.visits, 1)

    def ucb1(self, exploration=1.414):
        if self.visits == 0:
            return float('inf')
        exploit = self.mean_value
        explore = exploration * math.sqrt(
            math.log(self.parent.visits) / self.visits
        )
        return exploit + explore

def mcts_reasoning(model, prm, prompt, simulations=200,
                   tokens_per_step=256, max_depth=10):
    """MCTS over reasoning steps."""
    root = MCTSNode(tokens=tokenize(prompt))

    for _ in range(simulations):
        # Selection: traverse tree using UCB1
        node = root
        depth = 0
        while node.children and depth < max_depth:
            node = max(node.children, key=lambda c: c.ucb1())
            depth += 1

        # Expansion: generate a new reasoning step
        if depth < max_depth:
            new_step = model.generate(
                torch.tensor(node.tokens),
                max_new_tokens=tokens_per_step,
                temperature=0.8
            )
            child = MCTSNode(
                tokens=node.tokens + new_step.tolist(),
                parent=node
            )

            # Evaluate with PRM
            steps = segment_reasoning(detokenize(child.tokens))
            boundaries = compute_boundaries(prompt, steps)
            scores = prm(
                torch.tensor(child.tokens).unsqueeze(0),
                boundaries
            )
            child.prm_score = scores[0, -1].item()
            node.children.append(child)
            node = child

        # Backpropagation: update all ancestors
        value = node.prm_score
        while node is not None:
            node.visits += 1
            node.total_value += value
            node = node.parent

    # Return the path with highest visit count
    node = root
    while node.children:
        node = max(node.children, key=lambda c: c.visits)
    return detokenize(node.tokens)

The Cost Comparison

The headline economics: a 1B model with 10,000 thinking tokens versus a 405B model with 100 output tokens.

FLOPs Per Query

def flops_per_query(N_billions, prompt_tokens, output_tokens):
    """Total FLOPs for one query (prefill + decode)."""
    N = N_billions * 1e9
    prefill = 2 * N * prompt_tokens
    decode = 2 * N * output_tokens
    return prefill + decode

# 405B model, 500 prompt + 100 output
flops_405b = flops_per_query(405, 500, 100)
# = 2 * 405e9 * 600 = 4.86e14 FLOPs

# 1B model, 500 prompt + 10000 thinking + 100 output
flops_1b = flops_per_query(1, 500, 10100)
# = 2 * 1e9 * 10600 = 2.12e13 FLOPs

print(f"405B direct: {flops_405b:.2e} FLOPs")
print(f"1B + 10K think: {flops_1b:.2e} FLOPs")
print(f"Ratio: {flops_405b / flops_1b:.1f}x")
# 405B direct: 4.86e+14 FLOPs
# 1B + 10K think: 2.12e+13 FLOPs
# Ratio: 22.9x

The 1B model uses 22.9x fewer FLOPs. But FLOPs alone do not capture the full cost picture. We also need to account for the PRM inference cost and the memory/hardware differences.

Including PRM Cost

The PRM is typically 30-50% the size of the policy model. For a 1B policy, the PRM is approximately 0.5B parameters. PRM inference runs on each beam’s partial trace at each step.

def total_cost_with_prm(policy_B, prm_B, prompt_tokens, thinking_tokens,
                        beam_width=8, steps=20):
    """Total FLOPs including PRM scoring."""
    # Policy model: generate thinking_tokens across beam_width beams
    policy_flops = flops_per_query(policy_B, prompt_tokens, thinking_tokens)
    policy_flops *= beam_width  # Beam search generates beam_width traces

    # PRM: score each beam at each step
    avg_trace_length = prompt_tokens + thinking_tokens // 2  # Average length
    prm_flops_per_score = 2 * prm_B * 1e9 * avg_trace_length
    total_prm_scores = beam_width * steps
    prm_flops = prm_flops_per_score * total_prm_scores

    return policy_flops + prm_flops

cost_1b_with_prm = total_cost_with_prm(
    policy_B=1, prm_B=0.5, prompt_tokens=500,
    thinking_tokens=10000, beam_width=8, steps=20
)
cost_405b_direct = flops_per_query(405, 500, 100)

print(f"1B + PRM beam search: {cost_1b_with_prm:.2e} FLOPs")
print(f"405B direct: {cost_405b_direct:.2e} FLOPs")
print(f"Ratio: {cost_405b_direct / cost_1b_with_prm:.1f}x cheaper")

The 1B + PRM beam search configuration costs $0.11 per query — approximately 50x cheaper than the 405B model at$0.24 per query (which does not even include PRM or thinking). And it achieves 11.2 points higher accuracy.

When Small Models Do NOT Win

The “small model beats large model” result holds for structured reasoning tasks where the PRM can provide meaningful guidance. It breaks down in several regimes.

Knowledge-Intensive Tasks

Tasks requiring factual knowledge stored in parameters (trivia, world knowledge, niche domain expertise) favor large models regardless of thinking budget. A 1B model cannot reason its way to knowing that the Treaty of Westphalia was signed in 1648 — either the fact is in the weights or it is not.

Open-Ended Generation

Creative writing, summarization, and dialogue quality correlate with model size and do not benefit much from PRM-guided search. There is no “correct answer” to verify, so the PRM cannot provide useful signal.

Very Easy Tasks

If the base model already achieves high accuracy (over 90%), thinking tokens provide negligible improvement for both small and large models. The $\gamma \cdot \ln(T)$ term adds little when $Q_0$ is already near the ceiling.

Practical Deployment Strategy

The results suggest a tiered serving architecture:

class TieredInferenceRouter:
    """Route queries to the optimal model-thinking configuration."""

    def __init__(self):
        # Model pool
        self.models = {
            "1B": load_model("llama-1b"),
            "8B": load_model("llama-8b"),
            "70B": load_model("llama-70b"),
        }
        self.prm = load_prm("math-prm-500m")
        self.classifier = load_task_classifier()

    def route(self, query):
        task_type = self.classifier.classify(query)
        difficulty = self.classifier.difficulty(query)

        if task_type in ["math", "code", "logic"]:
            # Reasoning task: use small model + heavy thinking
            if difficulty == "hard":
                return self._serve_with_thinking(
                    "1B", query, thinking_budget=10000, beam_width=8
                )
            elif difficulty == "medium":
                return self._serve_with_thinking(
                    "1B", query, thinking_budget=2000, beam_width=4
                )
            else:
                return self._serve_with_thinking(
                    "1B", query, thinking_budget=500, beam_width=2
                )
        elif task_type in ["factual", "knowledge"]:
            # Knowledge task: use large model + minimal thinking
            return self._serve_direct("70B", query, max_tokens=500)
        else:
            # General task: medium model, no search
            return self._serve_direct("8B", query, max_tokens=1000)

    def _serve_with_thinking(self, model_name, query, thinking_budget, beam_width):
        model = self.models[model_name]
        return prm_beam_search(
            model, self.prm, query,
            beam_width=beam_width,
            max_steps=thinking_budget // 256,
            tokens_per_step=256,
        )

    def _serve_direct(self, model_name, query, max_tokens):
        model = self.models[model_name]
        return model.generate(query, max_new_tokens=max_tokens)

The tiered approach costs $0.018 per query (blended average) versus$0.24 for a single 405B model — 13x cheaper — while achieving higher quality on reasoning tasks. The key insight: no single model-size is optimal for all tasks. Small models with heavy thinking dominate reasoning. Large models with direct generation dominate knowledge. A router that understands the task type can exploit both regimes.

Summary

Test-time compute scaling inverts the traditional “bigger is better” paradigm for reasoning tasks. A 1B model with 10,000 thinking tokens and PRM-guided beam search outperforms a 405B model with direct generation on MATH-500 (73.6% vs 62.4%) at approximately 50x lower cost per query.

The mathematical explanation is the test-time scaling law: $Q(T) = Q_0 + \gamma \cdot \ln(T)$, where $\gamma$ is inversely related to model size. Smaller models have larger $\gamma$ because their errors are more amenable to correction through extended reasoning. The PRM is the critical enabler — its quality matters more than the policy model’s size.

The practical implications are architectural: deploy small models with co-located PRMs and generous thinking budgets for math, code, and logic tasks. Deploy large models with direct generation for knowledge-intensive tasks. A task-aware router that assigns queries to the appropriate configuration achieves better quality at lower cost than any single model size.