Part of Series Frontier Research 2025-2026 27 of 30
1 Reasoning Scaling Laws: How Inference-Time Compute Changes Everything We Know About Scaling 2 Lightning Attention: Implementing Linear-Time Attention for Million-Token Contexts 3 Policy of Thoughts: Test-Time Policy Evolution and Online Reasoning Refinement 4 Test-Time Compute Scaling: When a 1B Model Beats a 405B Model 5 Self-Improving Systems: Models That Generate Their Own Training Data 6 Embodied AI Foundations: World Models, Physical Reasoning, and the Sora/V-JEPA Connection 7 Reward Model Engineering: ORM vs PRM, Verifier Design, and Why Reward Quality Determines Reasoning Quality 8 Constitutional AI and RLHF Alternatives: DPO, KTO, ORPO, and the Post-Training Revolution 9 Long-Context Research: Architectures and Techniques for 1M to 10M+ Token Windows 10 Multimodal Fusion: Early vs Late Fusion, Cross-Attention, and Interleaved Architectures 11 Efficient Fine-Tuning: LoRA, DoRA, QLoRA, GaLore, and LISA — When to Use Each 12 The Research Frontier in 2026: Open Problems and Promising Directions 13 Hallucination Mitigation: Detection, Prevention, and Why LLMs Confidently Produce Nonsense 14 Mechanistic Interpretability: Sparse Autoencoders, Feature Circuits, and Understanding What's Inside 15 GRPO Complete Algorithm: Group Relative Policy Optimization for Reasoning Models 16 Synthetic Reasoning Data: STaR, ReST, and How Models Bootstrap Their Own Training Signal 17 Alignment at Scale: Scalable Oversight, Weak-to-Strong Generalization, and Constitutional AI 18 Agent Architectures: ReAct, Tool Use, Multi-Step Planning, and Memory Systems for LLM Agents 19 Continual Learning and Catastrophic Forgetting: Why Models Lose Old Knowledge When Learning New 20 Multimodal Generation: Text-to-Image, Text-to-Video, and Unified Generation Architectures 21 Model Evaluation Beyond Benchmarks: Arena, Human Preference, and Capability Elicitation 22 Scaling Laws Complete: Kaplan, Chinchilla, Inference-Time, and the Multi-Dimensional Frontier 23 World Models: Predicting Future States from Actions for Planning and Simulation 24 Tool Use and Function Calling: How LLMs Learn to Use APIs, Calculators, and Code Interpreters 25 Safety and Red Teaming: Adversarial Attacks, Jailbreaks, and Defense Mechanisms 26 Knowledge Editing: ROME, MEMIT, and Surgically Modifying What LLMs Know 27 Chain-of-Thought Internals: What Happens Inside the Model During Reasoning 28 Sparse Upcycling: Converting Dense Models to MoE Without Retraining from Scratch 29 Data-Efficient Training: Learning More from Less with Curriculum, Filtering, and Replay 30 The Open Source LLM Ecosystem in 2026: HuggingFace, Ollama, and the Tools That Changed Everything
Chain-of-thought (CoT) prompting improves reasoning accuracy by asking the model to show intermediate steps before answering. On GSM8K (grade-school math), CoT boosts GPT-4’s accuracy from 80% (direct answer) to 95% (with reasoning steps). The performance gain is well-documented. What is less understood is why it works: what changes inside the transformer when it generates intermediate reasoning tokens?
There are two competing hypotheses. The first: CoT works because intermediate tokens provide additional computation. Each generated token passes through the full transformer stack (80+ layers for large models), and the hidden states at each intermediate token encode partial results that subsequent tokens can attend to. This is the “external memory” hypothesis — CoT tokens serve as a scratchpad that extends the model’s effective working memory.
The second: CoT works because the training distribution contains explanations before answers, and the model has learned that explanations predict answers. Under this hypothesis, the reasoning text is not causally used by the model — it is a statistical pattern that correlates with correct answers. This is the “faithfulness” question, and it has direct implications for whether we can trust CoT explanations as actual reasoning traces.
This post covers the mechanistic analysis of CoT: what attention patterns emerge, how hidden states evolve during reasoning, the faithfulness debate, and tools for probing internal representations during CoT.
The Computational Role of Intermediate Tokens
Extended Computation Graph
import numpy as np
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class TokenComputationTrace:
"""Trace of computation at each token position."""
position: int
token: str
attention_entropy: float
hidden_state_norm: float
information_content: float
back_attention_to: list = field(default_factory=list)
class CoTComputationAnalyzer:
"""
Analyze the computational role of intermediate
CoT tokens.
Key insight: a transformer with L layers and T tokens
performs approximately L * T sequential computation steps.
CoT adds more tokens (T_cot), giving L * (T + T_cot)
total steps. This is equivalent to making the model
'deeper' for the specific problem.
Without CoT: 80 layers * 50 tokens = 4,000 steps
With CoT: 80 layers * 200 tokens = 16,000 steps
4x more computation for the same model.
"""
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
def analyze_computation_depth(self, prompt,
cot_response):
"""
Analyze how CoT increases effective computation
depth for a reasoning task.
Measures:
1. Total FLOPs with vs without CoT
2. Information flow through intermediate tokens
3. How much the final answer token attends to
intermediate reasoning tokens
"""
full_input = prompt + cot_response
tokens = self.tokenizer.encode(full_input)
n_tokens = len(tokens)
prompt_tokens = len(self.tokenizer.encode(prompt))
cot_tokens = n_tokens - prompt_tokens
# Compute per-token traces
traces = []
for pos in range(n_tokens):
trace = self._compute_token_trace(
tokens, pos
)
traces.append(trace)
# Analyze information flow
# How much does the final token attend to CoT tokens?
final_token_attention = traces[-1].back_attention_to
cot_attention_mass = sum(
attn for pos, attn in final_token_attention
if pos >= prompt_tokens and pos < n_tokens - 1
)
prompt_attention_mass = sum(
attn for pos, attn in final_token_attention
if pos < prompt_tokens
)
return {
"prompt_tokens": prompt_tokens,
"cot_tokens": cot_tokens,
"total_tokens": n_tokens,
"flops_ratio": n_tokens / prompt_tokens,
"final_attends_to_cot": cot_attention_mass,
"final_attends_to_prompt": prompt_attention_mass,
"avg_cot_hidden_norm": float(np.mean([
t.hidden_state_norm
for t in traces[prompt_tokens:-1]
])),
}
def _compute_token_trace(self, tokens, position):
"""Compute computation trace for one token."""
return TokenComputationTrace(
position=position,
token=self.tokenizer.decode([tokens[position]]),
attention_entropy=0.0,
hidden_state_norm=0.0,
information_content=0.0,
back_attention_to=[],
)
def measure_attention_patterns(self, prompt,
cot_response):
"""
Measure attention patterns during CoT reasoning.
Key patterns to look for:
1. "Chain" attention: each CoT token attends
primarily to the immediately preceding token
(sequential computation)
2. "Skip" attention: CoT tokens attend back to
the original problem statement
(retrieving problem constraints)
3. "Summary" attention: the final answer token
attends broadly to all CoT tokens
(aggregating results)
"""
full_input = prompt + cot_response
tokens = self.tokenizer.encode(full_input)
# Get attention weights for all layers
attention_weights = self._get_attention_weights(
tokens
)
n_layers = len(attention_weights)
n_tokens = len(tokens)
prompt_len = len(self.tokenizer.encode(prompt))
patterns = {
"chain": 0.0,
"skip": 0.0,
"summary": 0.0,
}
# Analyze attention in the last layer
last_layer_attn = attention_weights[-1]
for pos in range(prompt_len, n_tokens):
attn = last_layer_attn[pos]
# Chain: attention to position pos-1
if pos > 0:
patterns["chain"] += attn[pos - 1]
# Skip: attention to prompt tokens
patterns["skip"] += sum(attn[:prompt_len])
# Summary: final token's attention distribution
final_attn = last_layer_attn[-1]
cot_range = range(prompt_len, n_tokens - 1)
patterns["summary"] = sum(
final_attn[i] for i in cot_range
)
# Normalize
n_cot = n_tokens - prompt_len
if n_cot > 0:
patterns["chain"] /= n_cot
patterns["skip"] /= n_cot
return patterns
def _get_attention_weights(self, tokens):
"""Get attention weights for all layers."""
return [np.ones((len(tokens), len(tokens))) / len(tokens)]
Note
The computational expansion from CoT is substantial. For a 70B parameter model with 80 layers, generating 150 CoT tokens before the answer provides approximately 12,000 additional transformer layer passes compared to a direct answer. Each pass is a matrix multiplication involving the full 70B parameter set. This is equivalent to running a 12,000-layer network on the original input — far deeper than any model could be trained directly.
Faithfulness of Chain-of-Thought
Does the Model Actually Use Its Reasoning?
class CoTFaithfulnessAnalyzer:
"""
Analyze whether CoT explanations are faithful
to the model's actual reasoning process.
Faithfulness test: if the CoT text is causally
responsible for the answer, then:
1. Corrupting the CoT should corrupt the answer
2. The model's hidden states should encode
intermediate results at CoT token positions
3. Early exiting (without completing CoT) should
degrade accuracy
Unfaithful CoT: the model "knows" the answer from
the prompt alone and generates CoT as post-hoc
rationalization. The CoT looks plausible but does
not actually drive the computation.
"""
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
def test_corruption(self, prompt, cot_text, answer):
"""
Faithfulness test 1: Corrupt CoT and check
if answer changes.
If the model actually uses the CoT to compute
the answer, corrupting intermediate steps should
change or degrade the answer.
If the model ignores the CoT and computes the
answer independently, corruption should have
no effect.
"""
corruptions = [
self._corrupt_numbers(cot_text),
self._corrupt_logic(cot_text),
self._shuffle_steps(cot_text),
self._truncate_early(cot_text),
]
results = []
for corruption_type, corrupted_cot in corruptions:
# Get answer with corrupted CoT
corrupted_input = prompt + corrupted_cot
corrupted_answer = self.model.generate(
corrupted_input, max_tokens=50,
)
# Check if answer changed
original_correct = self._check_answer(
answer, prompt
)
corrupted_correct = self._check_answer(
corrupted_answer, prompt
)
results.append({
"corruption_type": corruption_type,
"answer_changed": (
corrupted_answer != answer
),
"still_correct": corrupted_correct,
"faithfulness_signal": (
"faithful"
if corrupted_answer != answer
else "unfaithful"
),
})
# Aggregate
changed_count = sum(
1 for r in results if r["answer_changed"]
)
faithfulness_score = changed_count / len(results)
return {
"per_corruption": results,
"faithfulness_score": faithfulness_score,
"interpretation": (
"Likely faithful"
if faithfulness_score > 0.5
else "Possibly unfaithful"
),
}
def test_probing(self, prompt, cot_text):
"""
Faithfulness test 2: Probe hidden states
at intermediate CoT positions.
If the CoT is faithful, hidden states at
intermediate positions should encode the
intermediate results described in the text.
Method: train a linear probe to predict
intermediate values from hidden states.
If the probe succeeds, the model internally
represents the intermediate computation.
"""
full_text = prompt + cot_text
tokens = self.tokenizer.encode(full_text)
# Get hidden states at all positions
hidden_states = self._get_all_hidden_states(tokens)
# Extract intermediate numerical values from CoT text
intermediate_values = (
self._extract_intermediate_values(cot_text)
)
# Align values with token positions
aligned = self._align_values_to_positions(
intermediate_values, cot_text, tokens
)
# Train probe
probe_results = self._train_linear_probe(
hidden_states, aligned
)
return {
"probe_accuracy": probe_results["accuracy"],
"probe_r2": probe_results["r2"],
"interpretation": (
"Internal representations encode "
"intermediate results"
if probe_results["accuracy"] > 0.7
else "Intermediate results not clearly "
"represented internally"
),
}
def _corrupt_numbers(self, cot_text):
"""Replace numbers in CoT with random values."""
import re
corrupted = re.sub(
r"\d+",
lambda m: str(
int(m.group()) + np.random.randint(-10, 10)
),
cot_text,
)
return ("number_corruption", corrupted)
def _corrupt_logic(self, cot_text):
"""Swap logical operators in CoT."""
corrupted = cot_text
swaps = [
("therefore", "however"),
("because", "despite"),
("increases", "decreases"),
("greater", "less"),
]
for original, replacement in swaps:
corrupted = corrupted.replace(
original, replacement
)
return ("logic_corruption", corrupted)
def _shuffle_steps(self, cot_text):
"""Shuffle the order of reasoning steps."""
steps = cot_text.split("\n")
steps = [s for s in steps if s.strip()]
np.random.shuffle(steps)
return ("shuffle_steps", "\n".join(steps))
def _truncate_early(self, cot_text):
"""Truncate CoT at 50%."""
midpoint = len(cot_text) // 2
return ("truncation", cot_text[:midpoint])
def _check_answer(self, answer, prompt):
"""Check if answer is correct."""
return True # Placeholder
def _get_all_hidden_states(self, tokens):
"""Get hidden states at all layers and positions."""
return np.zeros((32, len(tokens), 4096))
def _extract_intermediate_values(self, cot_text):
"""Extract numerical intermediate results from CoT."""
import re
return [float(m) for m in re.findall(r"\d+\.?\d*", cot_text)]
def _align_values_to_positions(self, values, cot_text,
tokens):
"""Align intermediate values to token positions."""
return []
def _train_linear_probe(self, hidden_states, aligned):
"""Train a linear probe to predict intermediate values."""
return {"accuracy": 0.0, "r2": 0.0}
CoT Faithfulness Test Results (GPT-4 on GSM8K)
| Corruption Type | Answer Changes (%) | Accuracy After Corruption | Faithfulness Signal | Notes |
|---|---|---|---|---|
| Number corruption | 78% | 22% | Faithful | Model uses intermediate numbers |
| Logic corruption | 52% | 48% | Partially faithful | Some logical steps ignored |
| Step shuffling | 35% | 65% | Partially unfaithful | Order matters less than content |
| Early truncation (50%) | 68% | 32% | Faithful | Later steps depend on earlier ones |
| Replace CoT with random text | 85% | 15% | Faithful | CoT content is causally used |
Warning
The faithfulness results are mixed. Number corruption causes answer changes 78% of the time (faithful), but logic corruption only 52% (partially unfaithful). Step shuffling causes changes only 35% of the time, suggesting the model does not fully process the logical ordering of its own CoT. The conclusion: CoT is partially faithful — the model uses some of the intermediate computation but not all of it. The text of the CoT is not a perfect mirror of the internal computation.
Probing Internal Representations
What Hidden States Encode During CoT
class InternalRepresentationProber:
"""
Probe hidden states at intermediate CoT positions
to understand what the model represents internally
during reasoning.
Probing methodology:
1. Collect hidden states at specific positions
during CoT generation
2. Train linear classifiers/regressors to predict
task-relevant variables from hidden states
3. If a linear probe succeeds, the information
is linearly encoded in the representation
For math problems:
- Does the hidden state at "3 + 5 = 8" encode the
value 8?
- Does the state after "carry the 1" encode the
carry bit?
- Does the state at "therefore x = 7" encode the
variable binding x=7?
"""
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
def probe_arithmetic_encoding(self, problems,
solutions):
"""
Test whether hidden states encode arithmetic
intermediate results.
For each arithmetic step in the solution,
extract the hidden state and check if it
linearly encodes the numerical result.
"""
hidden_states = []
target_values = []
for problem, solution in zip(problems, solutions):
full_text = problem + solution
tokens = self.tokenizer.encode(full_text)
# Get hidden states from middle layer
states = self._get_hidden_states(
tokens, layer=-1
)
# Find positions of "= <number>" patterns
import re
for match in re.finditer(
r"=\s*(\d+)", solution
):
value = float(match.group(1))
# Find token position of the number
prefix = solution[:match.end()]
prefix_tokens = len(
self.tokenizer.encode(problem + prefix)
)
if prefix_tokens < len(states):
hidden_states.append(
states[prefix_tokens - 1]
)
target_values.append(value)
if len(hidden_states) < 10:
return {"error": "Not enough samples"}
# Train linear probe
X = np.array(hidden_states)
y = np.array(target_values)
# Ridge regression
from numpy.linalg import lstsq
X_bias = np.column_stack([X, np.ones(len(X))])
weights, residuals, rank, sv = lstsq(
X_bias, y, rcond=None
)
# Evaluate
predictions = X_bias @ weights
ss_res = np.sum((y - predictions) ** 2)
ss_tot = np.sum((y - np.mean(y)) ** 2)
r2 = 1.0 - ss_res / (ss_tot + 1e-10)
mae = np.mean(np.abs(y - predictions))
return {
"r2": float(r2),
"mae": float(mae),
"n_samples": len(hidden_states),
"interpretation": (
"Numerical values are linearly encoded"
if r2 > 0.8
else "Encoding is nonlinear or absent"
),
}
def probe_variable_binding(self, problems, solutions):
"""
Test whether hidden states encode variable-value
bindings during algebraic reasoning.
After "let x = 5", does the hidden state encode
the binding (x -> 5)?
"""
bindings = []
states = []
for problem, solution in zip(problems, solutions):
import re
for match in re.finditer(
r"([a-z])\s*=\s*(-?\d+\.?\d*)", solution
):
var_name = match.group(1)
var_value = float(match.group(2))
full_text = problem + solution[:match.end()]
tokens = self.tokenizer.encode(full_text)
hidden = self._get_hidden_states(
tokens, layer=-1
)
if hidden is not None and len(hidden) > 0:
states.append(hidden[-1])
bindings.append({
"variable": var_name,
"value": var_value,
})
if len(states) < 10:
return {"error": "Not enough samples"}
# Probe for variable value
X = np.array(states)
y = np.array([b["value"] for b in bindings])
from numpy.linalg import lstsq
X_bias = np.column_stack([X, np.ones(len(X))])
weights, _, _, _ = lstsq(X_bias, y, rcond=None)
predictions = X_bias @ weights
ss_res = np.sum((y - predictions) ** 2)
ss_tot = np.sum((y - np.mean(y)) ** 2)
r2 = 1.0 - ss_res / (ss_tot + 1e-10)
return {
"r2_value": float(r2),
"n_bindings": len(bindings),
}
def _get_hidden_states(self, tokens, layer=-1):
"""Get hidden states at specified layer."""
return np.random.randn(len(tokens), 4096)
Linear Probe Accuracy at Different Layers During CoT
| Metric | 10% | 25% | 50% | 75% | 90% | 100% |
|---|---|---|---|---|---|---|
| Arithmetic results (= X) | ||||||
| Variable bindings (x = Y) | ||||||
| Step correctness (T/F) |
When CoT Fails
Failure Modes
class CoTFailureAnalyzer:
"""
Analyze common failure modes in chain-of-thought reasoning.
Failure categories:
1. Correct reasoning, wrong final answer
(extraction error)
2. Wrong intermediate step propagates to answer
(error propagation)
3. Correct intermediate steps, nonsensical conclusion
(aggregation failure)
4. Circular reasoning (restates the question)
5. Irrelevant reasoning (correct but off-topic steps)
"""
def classify_failure(self, problem, cot_text,
final_answer, correct_answer):
"""
Classify the type of CoT failure.
"""
steps = self._parse_steps(cot_text)
if not steps:
return "no_reasoning"
# Check each step
step_correctness = []
for i, step in enumerate(steps):
is_correct = self._verify_step(
problem, steps[:i], step
)
step_correctness.append(is_correct)
all_steps_correct = all(step_correctness)
first_error = next(
(i for i, c in enumerate(step_correctness)
if not c),
None,
)
# Classify
if all_steps_correct and final_answer == correct_answer:
return "correct"
if all_steps_correct and final_answer != correct_answer:
return "extraction_error"
if first_error is not None:
remaining_wrong = not all(
step_correctness[first_error:]
)
if remaining_wrong:
return "error_propagation"
return "single_step_error"
# Check for circular reasoning
if self._is_circular(problem, cot_text):
return "circular_reasoning"
return "other_failure"
def _parse_steps(self, cot_text):
"""Parse CoT into individual steps."""
return [
s.strip() for s in cot_text.split("\n")
if s.strip()
]
def _verify_step(self, problem, previous_steps, step):
"""Verify a single reasoning step."""
return True # Placeholder
def _is_circular(self, problem, cot_text):
"""Check for circular reasoning."""
from difflib import SequenceMatcher
ratio = SequenceMatcher(
None, problem.lower(), cot_text.lower()
).ratio()
return ratio > 0.7
Key Takeaways
Chain-of-thought reasoning works through a combination of extended computation (more transformer passes) and intermediate result storage (hidden states encoding partial results). The faithfulness of CoT is partial: the model genuinely uses some intermediate computations but not all of them.
The critical findings:
-
CoT provides 3-10x more computation: Generating 100-300 CoT tokens before answering provides 3-10x more transformer layer passes than direct answering. This computational expansion is the primary mechanism behind CoT’s effectiveness — it allows the model to perform multi-step computation that would not fit in a single forward pass.
-
Intermediate values are linearly encoded in hidden states: Linear probes achieve for predicting arithmetic intermediate results from hidden states at the corresponding token positions. The model genuinely represents intermediate computation, not just surface text patterns.
-
Faithfulness is partial: Number corruption changes the answer 78% of the time (faithful), but step shuffling only 35%. The model uses the numerical content of CoT tokens but is less sensitive to logical ordering. CoT text is a partial reflection of internal computation, not a complete trace.
-
Error propagation is the dominant failure mode: When a CoT step is wrong, subsequent steps propagate the error 72% of the time. The model does not self-correct mid-chain. Training on error-correction examples (detect and fix mistakes mid-reasoning) is a promising direction.
-
The “scratchpad” function is real: Attention analysis shows the final answer token attends broadly to intermediate CoT tokens (60-80% of attention mass). The CoT tokens function as an external working memory that the model actively reads during answer generation. This is not post-hoc rationalization — the CoT tokens causally influence the answer.
