Claude Architecture: Constitutional AI, RLHF at Scale, and the 200K Context Window

Claude 3.5 Sonnet refuses jailbreak attempts 98% of the time, versus GPT-4o’s 87%. The difference: Constitutional AI, where the model critiques its own outputs against a written constitution during RLHF training. Instead of hiring 1,000 human annotators to label harmful outputs, Anthropic wrote 75 constitutional principles and had Claude self-label 10 million outputs. The result is alignment that scales to the data, not to the annotation budget.

What Is Known About Claude’s Architecture

Published Details

Anthropic has been more transparent about their alignment methodology than about the base model architecture. What is confirmed:

• Dense architecture: Claude is a dense transformer (not MoE).
• Large parameter count: Exact numbers are not published, but Claude 3.5 Sonnet and Claude 3 Opus are frontier-quality, placing them in the 100B-300B+ range.
• 200K context window: Supported since Claude 3.
• Causal decoder-only: Standard autoregressive language model.
• Pre-norm: RMSNorm before attention and FFN sublayers.

# Inferred Claude architecture (based on Anthropic publications and behavior)
CLAUDE_INFERRED_CONFIG = {
    "architecture": "Dense causal decoder-only transformer",
    "normalization": "RMSNorm (pre-norm)",
    "attention": "GQA (likely, standard for dense models at this scale)",
    "ffn": "SwiGLU (likely, universal consensus)",
    "position_encoding": "RoPE with scaling for 200K context",
    "vocabulary": "BPE, 100K+ tokens (exact size unknown)",
    "context_length": 200000,
    "parameter_count": "Not disclosed",
}

What Differentiates Claude

Claude’s differentiation is not in the base architecture (which likely follows the standard dense transformer recipe) but in:

1. Constitutional AI for alignment
2. RLHF training methodology at scale
3. Extended context via continued pretraining
4. Safety properties integrated into the training pipeline

Constitutional AI (CAI)

The Problem CAI Solves

Standard RLHF requires human annotators to label harmful outputs. This creates several problems:

1. Exposure: Human labelers must read harmful content, which is psychologically damaging and creates liability.
2. Scale: Labeling harmful outputs for every possible harm category does not scale.
3. Consistency: Human judgments on what constitutes harm vary significantly across labelers.
4. Categories: New harm categories emerge faster than labeling protocols can be updated.

Constitutional AI addresses these by using the model itself to judge and revise harmful outputs, guided by a set of explicit principles (the “constitution”).

The CAI Pipeline

class ConstitutionalAIPipeline:
    """
    Constitutional AI training pipeline.
    Three phases: SL-CAI (supervised), RL-CAI (reinforcement learning),
    and constitution revision.
    """
    def __init__(self, base_model, constitution):
        self.base_model = base_model
        self.constitution = constitution
        self.revision_model = None
        self.reward_model = None

    def phase_1_generate_critiques(self, prompt, harmful_response):
        """
        Phase 1: Self-critique.
        The model critiques its own harmful response
        using constitutional principles.
        """
        critique_prompt = f"""
        Human: {prompt}

        Assistant response: {harmful_response}

        Please critique the above response according to the following principle:
        "{self.constitution['principles'][0]}"

        Identify specific problems and explain why the response violates this principle.
        """
        critique = self.base_model.generate(critique_prompt)
        return critique

    def phase_2_revise(self, prompt, harmful_response, critique):
        """
        Phase 2: Self-revision.
        The model revises its response to address the critique.
        """
        revision_prompt = f"""
        Human: {prompt}

        Original response: {harmful_response}

        Critique: {critique}

        Please revise the response to address the critique while remaining
        helpful and accurate. The revised response should follow this principle:
        "{self.constitution['principles'][0]}"
        """
        revised = self.base_model.generate(revision_prompt)
        return revised

    def phase_3_preference_data(self, prompt, original, revised):
        """
        Phase 3: Create preference pairs for RLHF.
        The revised response is preferred over the original.
        No human labeling of harmful content needed.
        """
        return {
            "prompt": prompt,
            "chosen": revised,      # Self-revised (constitutional)
            "rejected": original,   # Original (potentially harmful)
        }

The Constitution

The constitution is a set of explicit principles that define acceptable behavior. Examples from Anthropic’s published constitution:

EXAMPLE_CONSTITUTION = {
    "principles": [
        "Choose the response that is least likely to be harmful or dangerous "
        "to humans or the environment.",

        "Choose the response that is most respectful of everyone's privacy, "
        "autonomy, and rights.",

        "Choose the response that is most helpful while being honest and "
        "avoiding potential harms.",

        "Choose the response that demonstrates awareness of its own limitations "
        "and avoids overconfident claims.",

        "Choose the response that is least likely to be used for illegal "
        "or unethical purposes.",
    ],
    "source": "Drawn from UN Declaration of Human Rights, "
              "Anthropic's responsible AI principles, and best practices.",
}

SL-CAI: Supervised Learning Phase

def sl_cai_training(base_model, red_team_prompts, constitution):
    """
    SL-CAI (Supervised Learning from Constitutional AI).
    Creates a dataset of (prompt, revised_response) pairs.
    """
    training_pairs = []

    for prompt in red_team_prompts:
        # Step 1: Elicit a harmful response
        harmful = base_model.generate(
            prompt,
            system="You are a helpful assistant."  # No safety training yet
        )

        # Step 2: For each constitutional principle, critique and revise
        for principle in constitution["principles"]:
            critique = base_model.generate(
                f"Critique this response according to: {principle}\n"
                f"Response: {harmful}"
            )
            revised = base_model.generate(
                f"Revise this response to address: {critique}\n"
                f"Original: {harmful}\n"
                f"Principle: {principle}"
            )

        # Step 3: Use the final revision as the training target
        training_pairs.append({
            "prompt": prompt,
            "response": revised,  # Last revision is the target
        })

    # Step 4: Fine-tune the model on these pairs (standard SFT)
    sl_model = supervised_fine_tune(base_model, training_pairs)
    return sl_model

RL-CAI: Reinforcement Learning Phase

def rl_cai_training(sl_model, constitution, prompts):
    """
    RL-CAI: Train a reward model using AI-generated preferences,
    then use it for RLHF.
    """
    # Step 1: Generate pairs of responses
    preference_data = []
    for prompt in prompts:
        response_a = sl_model.generate(prompt, temperature=1.0)
        response_b = sl_model.generate(prompt, temperature=1.0)

        # Step 2: Use the constitution to choose the better response
        # The AI evaluates which response better follows the constitution
        evaluation = sl_model.generate(
            f"According to these principles: {constitution['principles']}\n"
            f"Which response is better?\n"
            f"Response A: {response_a}\n"
            f"Response B: {response_b}\n"
            f"Choose A or B and explain why."
        )

        chosen = response_a if "A" in evaluation[:10] else response_b
        rejected = response_b if "A" in evaluation[:10] else response_a

        preference_data.append({
            "prompt": prompt,
            "chosen": chosen,
            "rejected": rejected,
        })

    # Step 3: Train a reward model on these preferences
    reward_model = train_reward_model(preference_data)

    # Step 4: Use PPO or DPO to optimize the policy
    rl_model = rlhf_training(sl_model, reward_model, prompts)
    return rl_model

RLHF Training at Scale

The RLHF Pipeline

Anthropic’s RLHF pipeline is more extensive than most:

def anthropic_rlhf_pipeline():
    """
    Anthropic's RLHF training pipeline for Claude.
    """
    stages = {
        "pretraining": {
            "method": "Standard causal language modeling",
            "data": "Large internet corpus (exact composition unknown)",
            "tokens": "Not disclosed (likely 10T+)",
            "objective": "Next token prediction",
        },
        "sft_stage_1": {
            "method": "Supervised fine-tuning on helpful demonstrations",
            "data": "Human-written responses to diverse prompts",
            "volume": "Estimated 100K-1M examples",
            "objective": "Learn instruction-following behavior",
        },
        "sl_cai": {
            "method": "Supervised learning from Constitutional AI",
            "data": "Self-critiqued and revised responses",
            "volume": "Millions of examples (AI-generated)",
            "objective": "Learn safe behavior from constitutional principles",
        },
        "reward_modeling": {
            "method": "Train reward model from human + AI preferences",
            "data": "Mix of human preferences and AI-evaluated preferences",
            "volume": "Millions of preference pairs",
            "objective": "Score response quality (helpfulness + safety)",
        },
        "rl_optimization": {
            "method": "PPO or variant",
            "epochs": "Multiple rounds with updated reward models",
            "objective": "Maximize reward while maintaining coherence",
        },
        "red_teaming": {
            "method": "Adversarial testing by internal and external teams",
            "purpose": "Find remaining failure modes",
            "feedback": "Feeds back into next round of training",
        },
    }
    return stages

Reward Model Architecture

class RewardModel(nn.Module):
    """
    Reward model for RLHF.
    Takes a (prompt, response) pair and outputs a scalar reward.
    """
    def __init__(self, base_model_config):
        super().__init__()
        # Start from the same base model architecture
        self.transformer = TransformerDecoder(base_model_config)
        # Replace the language modeling head with a scalar head
        self.reward_head = nn.Linear(base_model_config.d_model, 1)

    def forward(self, input_ids, attention_mask):
        hidden_states = self.transformer(input_ids, attention_mask)
        # Use the last token's hidden state as the sequence representation
        last_hidden = hidden_states[:, -1, :]  # [B, d_model]
        reward = self.reward_head(last_hidden)  # [B, 1]
        return reward.squeeze(-1)

    def compute_loss(self, chosen_input_ids, rejected_input_ids, attention_masks):
        """
        Bradley-Terry loss: reward(chosen) should be higher than reward(rejected).
        """
        chosen_reward = self.forward(chosen_input_ids, attention_masks[0])
        rejected_reward = self.forward(rejected_input_ids, attention_masks[1])

        # Log-sigmoid loss
        loss = -torch.nn.functional.logsigmoid(chosen_reward - rejected_reward).mean()
        return loss, {
            "chosen_reward": chosen_reward.mean().item(),
            "rejected_reward": rejected_reward.mean().item(),
            "accuracy": (chosen_reward > rejected_reward).float().mean().item(),
        }

Multiple Reward Signals

Anthropic likely uses multiple reward models or a multi-dimensional reward:

def multi_reward_scoring():
    """
    Multiple reward dimensions for comprehensive evaluation.
    """
    reward_dimensions = {
        "helpfulness": {
            "description": "How well does the response answer the question?",
            "weight": 0.4,
            "source": "Human preference data",
        },
        "harmlessness": {
            "description": "Is the response free from harmful content?",
            "weight": 0.3,
            "source": "Constitutional AI + human red-teaming",
        },
        "honesty": {
            "description": "Does the response avoid false claims and express uncertainty appropriately?",
            "weight": 0.2,
            "source": "Factual verification + calibration data",
        },
        "coherence": {
            "description": "Is the response well-structured and logically consistent?",
            "weight": 0.1,
            "source": "Human quality judgments",
        },
    }
    return reward_dimensions

The 200K Context Window

Technical Approach

Claude 3 supports 200K tokens of context — more than Llama 3.1’s 128K but less than Gemini’s 1M. The likely implementation:

def claude_context_extension():
    """
    Inferred approach to 200K context in Claude.
    """
    approach = {
        "base_training_context": "8K-32K (typical for initial pretraining)",
        "extension_method": "RoPE scaling + continued pretraining",
        "rope_modification": "NTK-aware interpolation or YaRN",
        "continued_pretraining": "Additional training on long documents",
        "attention_optimization": "Flash Attention or custom efficient attention",
    }
    return approach

def rope_ntk_extension(base=10000, dim=128, original_ctx=8192, target_ctx=200000):
    """
    NTK-aware RoPE interpolation for context extension.
    """
    scale = target_ctx / original_ctx  # ~24.4x

    # NTK-aware: adjust the base frequency rather than linearly interpolating
    # This preserves high-frequency components (important for nearby tokens)
    # while stretching low-frequency components (for distant tokens)
    new_base = base * (scale ** (dim / (dim - 2)))

    freqs_original = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
    freqs_extended = 1.0 / (new_base ** (torch.arange(0, dim, 2).float() / dim))

    return {
        "original_base": base,
        "extended_base": new_base,
        "original_max_wavelength": (2 * torch.pi / freqs_original[-1]).item(),
        "extended_max_wavelength": (2 * torch.pi / freqs_extended[-1]).item(),
    }

Why 200K (Not 128K or 1M)

200K is a deliberate choice:

• More than 128K: Handles full codebases, long documents, and extensive conversation histories that 128K cannot.
• Less than 1M: Avoids the extreme compute cost of 1M context. Attention cost scales quadratically (or near-linearly with optimizations), and 200K is a practical sweet spot for most real-world applications.

def context_cost_analysis():
    """
    Compute cost as a function of context length.
    """
    context_lengths = [32000, 128000, 200000, 1000000]
    results = []

    for ctx in context_lengths:
        # Attention FLOPs: O(n^2 * d) per layer
        # With Flash Attention: still O(n^2) in FLOPs but O(n) in memory
        # Relative to 32K baseline
        relative_compute = (ctx / 32000) ** 2
        relative_memory = ctx / 32000  # KV cache scales linearly

        results.append({
            "context": ctx,
            "relative_compute": relative_compute,
            "relative_memory": relative_memory,
        })

    return results

Safety as Architecture

Integrated Safety Properties

Claude’s safety properties are not bolted on after training — they are integrated throughout the training pipeline. This produces qualitatively different behavior from models that rely on output filtering.

def safety_integration_levels():
    """
    Three levels of safety integration in LLMs.
    """
    levels = {
        "output_filtering": {
            "description": "Check outputs with a separate safety classifier",
            "robustness": "Low — adversarial prompts bypass easily",
            "latency_cost": "Adds inference latency",
            "examples": "Early ChatGPT, many open-source deployments",
        },
        "sft_safety": {
            "description": "Include safety examples in SFT data",
            "robustness": "Medium — learns surface patterns of safe responses",
            "latency_cost": "None",
            "examples": "Llama 3 Chat",
        },
        "constitutional_rlhf": {
            "description": "Reward model trained on constitutional principles, "
                          "multiple rounds of RL optimization",
            "robustness": "High — safety is in the model's reward function",
            "latency_cost": "None",
            "examples": "Claude",
        },
    }
    return levels

Red-Teaming Results

Anthropic publishes red-teaming results that demonstrate Claude’s safety properties:

What Makes Claude Competitive

Benchmark Performance

Despite fewer public architectural details than competitors, Claude performs at the frontier:

def claude_benchmark_performance():
    """
    Claude's benchmark performance across key tasks.
    """
    results = {
        "MMLU": {"Claude 3.5 Sonnet": 88.3, "GPT-4o": 88.7, "DeepSeek V3": 88.5},
        "GPQA": {"Claude 3.5 Sonnet": 59.4, "GPT-4o": 53.6, "DeepSeek V3": 59.1},
        "HumanEval": {"Claude 3.5 Sonnet": 92.0, "GPT-4o": 90.2, "DeepSeek V3": 92.7},
        "MATH-500": {"Claude 3.5 Sonnet": 78.3, "GPT-4o": 76.6, "DeepSeek V3": 90.2},
    }
    return results

Qualitative Strengths

Claude’s advantages are often in qualitative dimensions that benchmarks do not fully capture:

1. Instruction following: Claude is consistently rated as better at following complex, multi-part instructions.
2. Calibration: Claude is better at expressing uncertainty (“I’m not sure about this” when appropriate).
3. Safety: Refuses genuinely harmful requests while remaining helpful for legitimate edge cases.
4. Long-form writing: Quality of extended text generation (essays, reports, code) is consistently high.
5. Character consistency: Maintains a consistent persona and communication style across long conversations.

The Dense Architecture Choice

Why Not MoE?

Like Meta, Anthropic uses a dense architecture. The likely reasoning:

def dense_rationale_anthropic():
    """
    Why Anthropic chose dense over MoE for Claude.
    """
    reasons = {
        "alignment_properties": {
            "description": "Dense models have more predictable behavior. "
                          "MoE routing introduces stochasticity that can "
                          "complicate safety guarantees.",
            "importance": "Critical for Anthropic's safety-first mission",
        },
        "interpretability": {
            "description": "Dense models are easier to interpret. "
                          "Anthropic invests heavily in mechanistic interpretability. "
                          "MoE routing decisions add a layer of complexity.",
            "importance": "High — Anthropic's research agenda includes understanding models",
        },
        "serving_reliability": {
            "description": "Dense models have deterministic compute per token. "
                          "MoE models can have variable latency depending on "
                          "expert popularity.",
            "importance": "Important for API reliability",
        },
        "simpler_rlhf": {
            "description": "RLHF on MoE models is less studied. "
                          "Dense models have a more established RLHF pipeline.",
            "importance": "Moderate — Anthropic's RLHF pipeline is a key differentiator",
        },
    }
    return reasons

Anthropic’s Research Contributions

Published Research

Anthropic has published influential research that informs Claude’s development:

ANTHROPIC_KEY_PAPERS = {
    "Constitutional AI (2022)": {
        "contribution": "Self-supervised alignment without human harm labels",
        "impact": "Enabled scalable safety training",
        "used_in_claude": True,
    },
    "RLHF from AI Feedback (2022)": {
        "contribution": "Using AI feedback alongside human feedback",
        "impact": "Reduced human labeling requirements",
        "used_in_claude": True,
    },
    "Scaling Monosemanticity (2023)": {
        "contribution": "Dictionary learning for interpretable features",
        "impact": "Understanding what individual neurons compute",
        "used_in_claude": "Informs model development",
    },
    "Many-Shot Jailbreaking (2024)": {
        "contribution": "Demonstrated vulnerability of all LLMs to long-context attacks",
        "impact": "Improved long-context safety training",
        "used_in_claude": True,
    },
    "Sleeper Agents (2024)": {
        "contribution": "Showed that deceptive behavior can persist through safety training",
        "impact": "Motivated deeper alignment research",
        "used_in_claude": "Informs safety methodology",
    },
}

Mechanistic Interpretability

Anthropic’s interpretability research directly impacts Claude’s architecture:

def interpretability_impact():
    """
    How Anthropic's interpretability research affects Claude.
    """
    impacts = {
        "feature_detection": {
            "research": "Identifying meaningful features in transformer activations",
            "application": "Can detect when Claude is about to produce harmful content "
                          "by monitoring internal features",
        },
        "circuit_analysis": {
            "research": "Understanding computational circuits within transformers",
            "application": "Verify that safety behavior is implemented by stable circuits, "
                          "not surface-level pattern matching",
        },
        "probing": {
            "research": "Linear probes for internal knowledge representation",
            "application": "Detect when Claude 'knows' something it claims not to know "
                          "(honesty calibration)",
        },
    }
    return impacts

API and Serving Architecture

Claude’s API Design

Claude’s API exposes features that reflect the underlying architecture:

def claude_api_features():
    """
    Claude API features that reveal architectural capabilities.
    """
    features = {
        "system_prompt": {
            "description": "System prompt prepended to every conversation",
            "architectural_implication": "System prompt tokens are cached and reused",
        },
        "prompt_caching": {
            "description": "Cache long prefixes to reduce latency and cost",
            "architectural_implication": "KV cache can be stored and reused across requests",
            "cost_saving": "90% reduction for cached tokens",
        },
        "extended_thinking": {
            "description": "Model generates internal reasoning before responding",
            "architectural_implication": "Test-time compute scaling via chain of thought",
        },
        "tool_use": {
            "description": "Model can call external tools (search, code execution)",
            "architectural_implication": "Special token handling for tool calls",
        },
        "vision": {
            "description": "Process images alongside text",
            "architectural_implication": "Likely late-fusion (separate vision encoder)",
        },
    }
    return features

Prompt Caching

Prompt caching is a particularly interesting technical feature:

class PromptCacheManager:
    """
    Claude's prompt caching: store KV cache for common prefixes.
    """
    def __init__(self, max_cache_size_gb=100):
        self.cache = {}  # hash(prefix) -> KV cache
        self.max_size_gb = max_cache_size_gb

    def get_or_compute(self, prefix_tokens, model):
        """
        If prefix KV cache exists, return it.
        Otherwise, compute and cache.
        """
        prefix_hash = hash(tuple(prefix_tokens))

        if prefix_hash in self.cache:
            return self.cache[prefix_hash]  # Cache hit: skip prefix computation

        # Cache miss: compute KV cache for prefix
        kv_cache = model.compute_prefix_kv(prefix_tokens)
        self.cache[prefix_hash] = kv_cache
        return kv_cache

    def generate_with_cache(self, prefix_tokens, continuation_tokens, model):
        """
        Generate using cached prefix KV.
        Only needs to compute attention for new tokens,
        not re-process the entire prefix.
        """
        prefix_kv = self.get_or_compute(prefix_tokens, model)

        # Process only new tokens with prefix KV
        output = model.generate(
            continuation_tokens,
            past_key_values=prefix_kv,
        )
        return output

Extended Thinking

Test-Time Compute Scaling

Claude’s “extended thinking” feature allows the model to generate internal reasoning tokens before producing its final response. This is a form of test-time compute scaling.

def extended_thinking_analysis():
    """
    How extended thinking improves Claude's capabilities.
    """
    mechanism = {
        "how_it_works": "Claude generates a chain of thought (hidden from user) "
                        "before the final response. The hidden thinking can span "
                        "thousands of tokens for complex problems.",
        "when_it_helps": [
            "Multi-step mathematical proofs",
            "Complex code generation",
            "Logical reasoning problems",
            "Ambiguous instructions requiring careful interpretation",
        ],
        "quality_improvement": {
            "MATH-500": "+10-15 points with extended thinking",
            "coding_problems": "+5-10% pass rate",
            "complex_instructions": "Significantly better adherence",
        },
        "cost_tradeoff": "Uses more tokens (higher cost and latency) "
                         "but dramatically improves accuracy on hard problems",
    }
    return mechanism

Claude Model Family

Summary

Claude’s architecture is likely a standard dense transformer, but its differentiation comes from the training methodology:

• Constitutional AI: Self-supervised alignment that scales without exposing human labelers to harmful content. The constitution provides explicit, updatable principles.
• RLHF at scale: Multiple rounds of reward modeling and policy optimization with both human and AI feedback.
• 200K context: Practical balance between capability and cost, covering 99%+ of real-world use cases.
• Dense architecture: Enables better interpretability, more predictable behavior, and simpler RLHF — aligned with Anthropic’s safety-first mission.
• Prompt caching: Architectural optimization for API efficiency, reducing latency and cost for common usage patterns.
• Extended thinking: Test-time compute scaling that dramatically improves performance on complex reasoning tasks.

Claude’s approach demonstrates that at the frontier, how you train the model (alignment methodology, reward engineering, safety integration) matters as much as the base architecture. Two models with identical base architectures can have dramatically different safety, truthfulness, and instruction-following properties based on post-training alone.