Phi and Small Language Models: How Microsoft Achieves GPT-3.5 Quality at 3B Parameters

Microsoft’s Phi series proved a controversial claim: data quality matters more than model size. Phi-1 (1.3B) matched models 10x its size on code benchmarks. Phi-3-mini (3.8B) approached GPT-3.5 across general benchmarks. Phi-4 pushed the frontier further. This post covers the data pipeline, the architectural decisions (surprisingly standard), the scaling law argument, and the practical implications.

The Phi Timeline

Evolution

def phi_model_timeline():
    """
    The Phi series: progressively stronger small models.
    Key theme: architecture barely changes, data gets dramatically better.
    """

    models = {
        'phi_1': {
            'release': 'June 2023',
            'params': '1.3B',
            'architecture': 'Standard transformer (24 layers, 2048 d_model)',
            'training_data': 'Textbook-quality synthetic code data',
            'training_tokens': '~7B',
            'key_result': 'HumanEval 50.6% (matches Llama 65B: ~47%)',
            'key_innovation': 'Synthetic "textbook" data for code',
        },
        'phi_1_5': {
            'release': 'September 2023',
            'params': '1.3B',
            'architecture': 'Same as Phi-1',
            'training_data': 'Textbook-quality synthetic + filtered web data',
            'training_tokens': '~30B',
            'key_result': 'Common sense reasoning comparable to 5x larger models',
            'key_innovation': 'Extended textbook approach to general knowledge',
        },
        'phi_2': {
            'release': 'December 2023',
            'params': '2.7B',
            'architecture': 'Standard transformer (32 layers, 2560 d_model)',
            'training_data': 'Synthetic textbooks + filtered web + code',
            'training_tokens': '250B',
            'key_result': 'Matches Llama 2 70B on some benchmarks',
            'key_innovation': 'Knowledge transfer from larger models',
        },
        'phi_3_mini': {
            'release': 'April 2024',
            'params': '3.8B',
            'architecture': 'Standard transformer with grouped query attention',
            'training_data': 'Heavily filtered web + synthetic + post-training',
            'training_tokens': '3.3T',
            'key_result': 'Approaches GPT-3.5 Turbo across benchmarks',
            'key_innovation': 'Massive token count on curated data',
        },
        'phi_3_medium': {
            'release': 'May 2024',
            'params': '14B',
            'architecture': 'Standard transformer with GQA',
            'training_data': 'Same pipeline as Phi-3-mini, more capacity',
            'training_tokens': '4.8T',
            'key_result': 'Approaches GPT-4 on some benchmarks',
            'key_innovation': 'Scaling up while maintaining data quality',
        },
        'phi_4': {
            'release': 'December 2024',
            'params': '14B',
            'architecture': 'Standard transformer with GQA',
            'training_data': 'Synthetic data pivoting from web-centric to quality-centric',
            'training_tokens': '9.8T',
            'key_result': 'Exceeds GPT-4o on several benchmarks',
            'key_innovation': 'Pivot from web data to diverse synthetic generation',
        },
    }
    return models

The Data Pipeline: Textbook Quality at Scale

What “Textbook Quality” Means

def textbook_quality_definition():
    """
    Microsoft defines "textbook quality" data as content that:
    1. Is self-contained (no external dependencies to understand)
    2. Progresses from simple to complex
    3. Includes explanations, not just facts
    4. Uses consistent notation and terminology
    5. Avoids noise (ads, navigation, boilerplate)
    """

    # Example: BAD web data (noisy, fragmented)
    bad_example = """
    Posted by user123 on StackOverflow:
    How do I sort a list in Python? I tried sort() but it doesn't work.

    Answer by moderator (accepted, 453 upvotes):
    Use sorted(). Example: sorted_list = sorted(my_list)
    You can also use list.sort() which sorts in-place.

    [ADVERTISEMENT: Learn Python at CodeCamp — 50% off this week!]
    """

    # Example: GOOD textbook-quality data (clean, educational)
    good_example = """
    ## Sorting Algorithms in Python

    Python provides two built-in approaches for sorting sequences.

    The sorted() function creates a new sorted list from any iterable:

    ```python
    numbers = [3, 1, 4, 1, 5, 9, 2, 6]
    sorted_numbers = sorted(numbers)
    # sorted_numbers = [1, 1, 2, 3, 4, 5, 6, 9]
    # numbers is unchanged

The list.sort() method sorts a list in-place, modifying the original list and returning None:

numbers = [3, 1, 4, 1, 5, 9, 2, 6]
numbers.sort()
# numbers is now [1, 1, 2, 3, 4, 5, 6, 9]

Both use Timsort (O(n log n) average case), which is a hybrid of merge sort and insertion sort. For custom ordering, pass a key function:

words = ["banana", "apple", "cherry"]
sorted_by_length = sorted(words, key=len)
# ["apple", "banana", "cherry"]

"""

return bad_example, good_example

### Synthetic Data Generation Pipeline

```python
class PhiDataPipeline:
    """
    Phi's data pipeline generates synthetic textbook-quality data
    using a larger teacher model (e.g., GPT-4).
    """

    def __init__(self, teacher_model, topics, config):
        self.teacher = teacher_model
        self.topics = topics
        self.config = config

    def generate_textbook_page(self, topic, subtopic, difficulty):
        """
        Generate one "page" of textbook-quality content.
        """
        prompt = f"""Write a clear, educational explanation of {subtopic}
in the context of {topic}. The explanation should be at a
{difficulty} level. Include:

1. A concise definition or introduction
2. A concrete code example (if applicable) or worked example
3. An explanation of WHY this works, not just HOW
4. A common mistake or edge case

Write in a textbook style: formal but accessible.
Do not reference external links, other chapters, or prerequisites.
The text should be completely self-contained."""

        response = self.teacher.generate(
            prompt,
            temperature=0.7,
            max_tokens=2048,
        )
        return response

    def generate_exercise(self, topic, concept):
        """
        Generate a practice exercise with solution.
        """
        prompt = f"""Create a programming exercise about {concept} in {topic}.
Include:
1. Problem statement (2-3 sentences)
2. Input/output examples
3. A complete solution with comments
4. An explanation of the approach"""

        return self.teacher.generate(prompt, temperature=0.8, max_tokens=1024)

    def quality_filter(self, text):
        """
        Filter generated content for quality.
        Multiple criteria must pass.
        """
        checks = {
            'length': len(text.split()) > 100,
            'has_code': '```' in text or 'def ' in text or 'class ' in text,
            'no_refs': 'click here' not in text.lower() and 'see chapter' not in text.lower(),
            'no_errors': self._check_code_compiles(text),
            'educational': self._check_educational_markers(text),
        }

        # All checks must pass
        return all(checks.values())

    def _check_code_compiles(self, text):
        """Extract code blocks and verify they parse."""
        import re
        code_blocks = re.findall(r'```python\n(.*?)```', text, re.DOTALL)
        for code in code_blocks:
            try:
                compile(code, '<string>', 'exec')
            except SyntaxError:
                return False
        return True

    def _check_educational_markers(self, text):
        """Check for educational structure markers."""
        markers = ['example', 'note', 'important', 'because', 'why', 'how']
        text_lower = text.lower()
        return sum(1 for m in markers if m in text_lower) >= 2

    def generate_curriculum(self, num_pages=1000000):
        """
        Generate a full curriculum across all topics.
        """
        pages = []
        for topic in self.topics:
            subtopics = self._get_subtopics(topic)
            for subtopic in subtopics:
                for difficulty in ['introductory', 'intermediate', 'advanced']:
                    page = self.generate_textbook_page(topic, subtopic, difficulty)
                    if self.quality_filter(page):
                        pages.append(page)

                    if len(pages) >= num_pages:
                        return pages

        return pages

    def _get_subtopics(self, topic):
        """Generate subtopics for a topic using the teacher model."""
        prompt = f"List 20 important subtopics in {topic}, one per line."
        response = self.teacher.generate(prompt, temperature=0.3, max_tokens=500)
        return [line.strip() for line in response.split('\n') if line.strip()]

Architecture: Deliberately Standard

Phi’s Architectural Choices

class Phi3MiniConfig:
    """
    Phi-3-mini architecture: a standard transformer.
    The architecture is NOT the innovation. The data is.
    """

    def __init__(self):
        # Standard transformer
        self.d_model = 3072
        self.num_layers = 32
        self.num_heads = 32
        self.head_dim = 96
        self.d_ff = 8192

        # Grouped Query Attention (standard, not novel)
        self.num_kv_heads = 8
        self.gqa_ratio = 4  # 32 / 8

        # Positional encoding
        self.rope = True
        self.max_position = 4096  # Extended to 128K with LongRoPE

        # Vocabulary
        self.vocab_size = 32064

        # Activation
        self.activation = 'silu'  # SwiGLU FFN

        # Total parameters
        self.total_params = 3.8e9

        # What is NOT special about this architecture:
        self.not_novel = [
            "No MoE",
            "No MLA (Multi-head Latent Attention)",
            "No multi-token prediction",
            "No novel attention pattern",
            "No novel positional encoding (standard RoPE)",
            "Standard SwiGLU FFN",
            "Standard GQA (same as Llama 3)",
        ]

def architecture_comparison():
    """
    Compare Phi-3-mini architecture with peers.
    Key point: architecturally identical to Llama 3 at smaller scale.
    """

    models = {
        'phi_3_mini': {
            'params': '3.8B',
            'd_model': 3072,
            'layers': 32,
            'heads': 32,
            'kv_heads': 8,
            'gqa': True,
            'activation': 'SwiGLU',
            'rope': True,
            'moe': False,
        },
        'llama_3_8B': {
            'params': '8B',
            'd_model': 4096,
            'layers': 32,
            'heads': 32,
            'kv_heads': 8,
            'gqa': True,
            'activation': 'SwiGLU',
            'rope': True,
            'moe': False,
        },
        'gemma_2B': {
            'params': '2B',
            'd_model': 2048,
            'layers': 18,
            'heads': 8,
            'kv_heads': 1,
            'gqa': True,
            'activation': 'GeGLU',
            'rope': True,
            'moe': False,
        },
    }

    # Phi-3-mini is architecturally a smaller Llama 3.
    # The ONLY difference is the training data.
    return models

The Scaling Law Argument

Chinchilla vs Phi

def scaling_law_analysis():
    """
    Chinchilla scaling law: optimal performance requires
    ~20 tokens per parameter (approximately).

    Phi violates this dramatically:
    - Phi-3-mini: 3.8B params, 3.3T tokens = 868 tokens per param
    - Chinchilla optimal: 3.8B params * 20 = 76B tokens

    Phi uses 43x more tokens than Chinchilla optimal.
    This works because the DATA QUALITY is much higher.
    """

    models = {
        'chinchilla_70B': {
            'params_B': 70,
            'tokens_T': 1.4,
            'tokens_per_param': 20,
            'approach': 'Chinchilla-optimal compute allocation',
        },
        'llama_2_7B': {
            'params_B': 7,
            'tokens_T': 2.0,
            'tokens_per_param': 286,
            'approach': 'Over-train for inference efficiency',
        },
        'llama_3_8B': {
            'params_B': 8,
            'tokens_T': 15.0,
            'tokens_per_param': 1875,
            'approach': 'Extreme over-training',
        },
        'phi_3_mini': {
            'params_B': 3.8,
            'tokens_T': 3.3,
            'tokens_per_param': 868,
            'approach': 'Over-train on curated data',
        },
        'phi_4': {
            'params_B': 14,
            'tokens_T': 9.8,
            'tokens_per_param': 700,
            'approach': 'Heavy synthetic data',
        },
    }

    return models

def data_quality_scaling():
    """
    The Phi argument: quality-adjusted token count matters more
    than raw token count.

    If "textbook quality" data is worth 10x web data in terms
    of learning efficiency, then:
    - 3.3T textbook tokens ~= 33T web tokens in learning value
    - This would make Phi-3-mini's effective training equivalent
      to a model trained on 33T web tokens
    """

    quality_multipliers = {
        'raw_web': {
            'quality_multiplier': 1.0,
            'examples': 'Common Crawl, unfiltered Reddit',
            'noise_level': 'High — ads, spam, errors, duplicates',
        },
        'filtered_web': {
            'quality_multiplier': 2.0,
            'examples': 'RefinedWeb, C4',
            'noise_level': 'Medium — basic dedup and filtering',
        },
        'curated_web': {
            'quality_multiplier': 5.0,
            'examples': 'Wikipedia, StackOverflow (accepted answers)',
            'noise_level': 'Low — community-moderated',
        },
        'textbook_synthetic': {
            'quality_multiplier': 10.0,
            'examples': 'Phi synthetic data, textbook-quality explanations',
            'noise_level': 'Very low — generated with quality constraints',
        },
    }

    # Phi-3-mini effective training:
    # 3.3T textbook tokens * 10x quality multiplier = 33T effective tokens
    # Equivalent to training a 33T-token model on web data
    # At 3.8B params, this is 8684 effective tokens per parameter

    return quality_multipliers

Training Details

Multi-Phase Training

def phi3_training_phases():
    """
    Phi-3-mini uses a multi-phase training approach.
    """

    phases = {
        'phase_1_pretraining': {
            'data': 'Mix of web (filtered) + synthetic textbook data',
            'tokens': '~3.3T',
            'lr': 3e-4,
            'batch_size': '4M tokens',
            'context_length': 4096,
            'optimizer': 'AdamW (beta1=0.9, beta2=0.95)',
            'warmup': '2000 steps',
            'decay': 'Cosine to 3e-5',
        },
        'phase_2_long_context': {
            'data': 'Long documents, interleaved short + long sequences',
            'tokens': '~500B',
            'context_length': '4096 -> 128K (LongRoPE extension)',
            'method': 'Progressive context extension',
            'lr': '1e-5 (lower to preserve knowledge)',
        },
        'phase_3_post_training': {
            'data': 'SFT on instruction-following data',
            'method': 'Supervised fine-tuning + DPO alignment',
            'tokens': '~10B',
            'lr': '2e-5',
        },
    }
    return phases

def data_mixing_strategy():
    """
    Phi-3-mini data mixing during pre-training.
    """

    mix = {
        'synthetic_textbook': {
            'fraction': 0.35,
            'source': 'GPT-4 generated textbook-style content',
            'topics': 'CS, math, science, reasoning, general knowledge',
            'quality': 'Highest — filtered and verified',
        },
        'synthetic_exercises': {
            'fraction': 0.15,
            'source': 'GPT-4 generated coding exercises with solutions',
            'quality': 'High — code verified to compile and pass tests',
        },
        'filtered_web': {
            'fraction': 0.30,
            'source': 'Web data filtered by educational quality classifier',
            'filtering': 'Keep only content that resembles textbook material',
            'quality': 'Medium-high — automated filtering',
        },
        'code': {
            'fraction': 0.15,
            'source': 'GitHub code (deduplicated, filtered for quality)',
            'quality': 'Medium — standard code filtering',
        },
        'other': {
            'fraction': 0.05,
            'source': 'Books, Wikipedia, other curated sources',
            'quality': 'High — manually curated',
        },
    }

    # Key insight: 50% of the training data is SYNTHETIC
    # This is much higher than any other frontier model
    synthetic_fraction = mix['synthetic_textbook']['fraction'] + mix['synthetic_exercises']['fraction']
    # 0.35 + 0.15 = 0.50 = 50%

    return mix, synthetic_fraction

The Educational Quality Classifier

class EducationalQualityClassifier:
    """
    Classifier used to filter web data for educational quality.
    Trained to identify "textbook-like" web pages.
    """

    def __init__(self, model_path):
        self.model = self._load_classifier(model_path)

    def score(self, text):
        """
        Score text on educational quality [0, 1].
        1.0 = perfect textbook quality
        0.0 = noise/spam/irrelevant
        """
        features = self._extract_features(text)
        return self.model.predict_proba(features)[0][1]

    def _extract_features(self, text):
        """
        Features that correlate with educational quality:
        """
        words = text.split()
        sentences = text.split('.')

        features = {
            # Structural features
            'has_code_blocks': int('```' in text or '    ' in text),
            'has_headers': int('#' in text or text.count('\n\n') > 3),
            'has_lists': int('1.' in text or '- ' in text),
            'has_examples': int('example' in text.lower() or 'e.g.' in text.lower()),

            # Content features
            'avg_sentence_length': (
                sum(len(s.split()) for s in sentences) / (len(sentences) + 1)
            ),
            'unique_word_ratio': len(set(words)) / (len(words) + 1),
            'technical_term_density': self._count_technical_terms(text) / (len(words) + 1),

            # Quality indicators
            'has_explanation_markers': int(any(
                m in text.lower() for m in ['because', 'therefore', 'this means',
                                             'in other words', 'note that']
            )),
            'has_noise_markers': int(any(
                m in text.lower() for m in ['click here', 'subscribe', 'cookie',
                                             'advertisement', 'sign up']
            )),

            # Length features
            'word_count': len(words),
            'paragraph_count': text.count('\n\n') + 1,
        }
        return features

    def _count_technical_terms(self, text):
        technical_terms = [
            'function', 'algorithm', 'variable', 'parameter', 'return',
            'class', 'method', 'object', 'array', 'list', 'dictionary',
            'theorem', 'proof', 'equation', 'matrix', 'vector',
        ]
        text_lower = text.lower()
        return sum(text_lower.count(term) for term in technical_terms)

    def _load_classifier(self, path):
        pass  # Load trained classifier

def filtering_impact():
    """
    Impact of quality filtering on training data.
    """

    filtering_results = {
        'raw_web_pages': 1_000_000_000,        # 1B pages
        'after_dedup': 400_000_000,             # 400M pages
        'after_language_filter': 200_000_000,   # 200M pages
        'after_quality_score_0.5': 20_000_000,  # 20M pages (10%)
        'after_quality_score_0.8': 5_000_000,   # 5M pages (2.5%)

        # Only 2.5% of web data meets the quality threshold
        # But this 2.5% is worth more than the other 97.5%
        'pass_rate': '2.5%',
        'quality_vs_quantity': 'Phi proves 2.5% of web data + synthetic > 100%',
    }
    return filtering_results

Phi-4: The Synthetic Data Pivot

def phi4_innovations():
    """
    Phi-4 (December 2024) represents a shift in the Phi approach.
    """

    phi4_details = {
        'architecture': {
            'params': '14B',
            'd_model': 5120,
            'layers': 40,
            'heads': 40,
            'kv_heads': 10,
            'context_length': 16384,
            'note': 'Still standard transformer — no MoE, no MLA',
        },
        'data_pivot': {
            'old_approach': 'Synthetic textbook data + filtered web data',
            'new_approach': 'Synthetic data as PRIMARY source, web data as supplement',
            'synthetic_fraction': '~70% of total tokens are synthetic',
            'key_change': 'Synthetic data generators trained on diverse seed topics',
        },
        'training': {
            'total_tokens': '9.8T',
            'tokens_per_param': 700,
            'multi_stage': True,
            'post_training': 'SFT + DPO + GRPO (borrowed from DeepSeek-R1)',
        },
        'results': {
            'MMLU': 84.8,
            'GPQA': 56.1,
            'MATH': 80.4,
            'HumanEval': 82.6,
            'note': 'Exceeds GPT-4o on MATH (74.6%) and competitive elsewhere',
        },
    }
    return phi4_details

def synthetic_data_scaling():
    """
    Phi-4's synthetic data scaling strategy.
    """

    strategy = {
        'seed_diversity': {
            'description': 'Use a diverse set of seed topics to generate data',
            'implementation': 'Topic taxonomy with 10K+ leaf topics',
            'importance': 'Prevents synthetic data collapse (all data looking similar)',
        },
        'multi_teacher': {
            'description': 'Use multiple teacher models for generation',
            'implementation': 'GPT-4, GPT-4o, Claude, Gemini as teachers',
            'importance': 'Each teacher has different strengths/biases',
        },
        'quality_verification': {
            'description': 'Verify generated content with execution and grading',
            'implementation': 'Run code, check math, grade essay quality',
            'importance': 'Catch teacher model errors before training',
        },
        'decontamination': {
            'description': 'Remove benchmark-similar content from training data',
            'implementation': 'N-gram overlap detection against benchmark sets',
            'importance': 'Ensure benchmark results are genuine, not memorized',
        },
    }
    return strategy

Criticisms and Limitations

def phi_criticisms():
    """
    The Phi series faces several legitimate criticisms.
    """

    criticisms = {
        'benchmark_contamination_concern': {
            'criticism': 'Synthetic data generated by GPT-4 may contain '
                        'patterns similar to benchmark questions',
            'response': 'Microsoft claims aggressive decontamination. '
                       'Independent verification is limited.',
            'validity': 'Partially valid — contamination is hard to fully eliminate',
        },
        'narrow_capabilities': {
            'criticism': 'Phi models excel at benchmarks but may struggle '
                        'with open-ended, creative, or conversational tasks',
            'response': 'Phi-3 and Phi-4 include post-training for chat. '
                       'General quality has improved.',
            'validity': 'Partially valid — small models have less "world knowledge"',
        },
        'cost_of_synthetic_data': {
            'criticism': 'Generating 50% of training data with GPT-4 is expensive. '
                        'The total cost of data generation may approach '
                        'the cost of training a larger model on web data.',
            'response': 'Data generation is a one-time cost that amortizes '
                       'across multiple model trainings.',
            'validity': 'Valid for one-off training. Less valid if data is reused.',
        },
        'reproducibility': {
            'criticism': 'Microsoft has not released the training data. '
                        'Others cannot reproduce the results.',
            'response': 'Model weights are open. The data pipeline '
                       'methodology is described in papers.',
            'validity': 'Valid — without data, true reproduction is impossible',
        },
        'scaling_ceiling': {
            'criticism': 'Data quality gains may plateau. Eventually model size '
                        'matters for storing more knowledge.',
            'response': 'Phi-4 at 14B shows continued improvement. '
                       'The ceiling is not yet reached.',
            'validity': 'Unknown — too early to tell',
        },
    }
    return criticisms

Practical Implications

When to Use Phi-Class Models

def deployment_analysis():
    """
    Phi models are ideal for specific deployment scenarios.
    """

    good_fit = {
        'edge_deployment': {
            'scenario': 'Running on phone, laptop, or edge device',
            'why_phi': '3.8B fits in 4-8 GB RAM (quantized). '
                      'No other model this size has comparable quality.',
            'example': 'On-device code completion, math tutoring',
        },
        'cost_sensitive_apis': {
            'scenario': 'Serving millions of simple queries cheaply',
            'why_phi': '3.8B inference is 10-50x cheaper than 70B models. '
                      'Quality is sufficient for many tasks.',
            'example': 'Customer support, content classification',
        },
        'fine_tuning_base': {
            'scenario': 'Need a small model fine-tuned for a specific domain',
            'why_phi': 'Better starting point than other 3-7B models. '
                      'Strong reasoning foundation transfers well.',
            'example': 'Medical QA, legal document analysis',
        },
        'latency_critical': {
            'scenario': 'Need sub-100ms response time',
            'why_phi': 'Small model = fewer FLOPs per token = lower latency. '
                      'Quality comparable to 10x larger models.',
            'example': 'Real-time coding assistance, chatbots',
        },
    }

    bad_fit = {
        'knowledge_intensive': {
            'scenario': 'Need broad, deep world knowledge',
            'why_not_phi': '3.8B cannot store as much factual knowledge as 70B+. '
                          'Will hallucinate more on obscure topics.',
            'example': 'General-purpose assistant, encyclopedia queries',
        },
        'long_context_reasoning': {
            'scenario': 'Need to reason over 100K+ token contexts',
            'why_not_phi': 'Small models have limited attention capacity. '
                          'Long-context quality degrades faster than large models.',
            'example': 'Analyzing entire codebases, long document QA',
        },
        'multilingual': {
            'scenario': 'Need strong performance in many languages',
            'why_not_phi': 'Phi is trained primarily on English. '
                          'Multilingual performance is weaker than Llama 3 or Qwen.',
            'example': 'Global customer support in 50 languages',
        },
    }

    return good_fit, bad_fit

The Broader Lesson

def key_takeaways():
    """
    What the Phi series teaches the field.
    """

    lessons = {
        'data_over_architecture': {
            'lesson': 'Standard transformer + excellent data beats novel '
                     'architecture + mediocre data, at small scale.',
            'evidence': 'Phi-3-mini uses identical architecture to Llama 3 '
                       'but matches models 2-5x its size.',
        },
        'synthetic_data_works': {
            'lesson': 'High-quality synthetic data is a viable training strategy.',
            'evidence': '50%+ synthetic data in Phi-3/4 with strong results. '
                       'No evidence of "model collapse" concerns at this scale.',
        },
        'over_training_is_optimal_for_deployment': {
            'lesson': 'Training small models on many more tokens than '
                     'Chinchilla-optimal produces better per-FLOP-of-inference quality.',
            'evidence': 'Phi-3-mini at 868 tokens/param significantly '
                       'outperforms Chinchilla-optimal allocation.',
        },
        'size_still_matters_eventually': {
            'lesson': 'Data quality cannot infinitely compensate for model size. '
                     'At some point, you need more parameters for more knowledge.',
            'evidence': 'Phi-3-mini still trails GPT-4 on knowledge-intensive benchmarks. '
                       'Phi-4 at 14B closes more of this gap.',
        },
    }
    return lessons

The Phi series forces a reconsideration of scaling laws. The standard view — “bigger is better” — is correct only when data quality is held constant. When data quality improves, the size-quality relationship shifts dramatically. A 3.8B model trained on carefully curated data achieves what was previously thought to require 20-70B parameters. This does not invalidate scaling laws; it adds a new dimension — data quality — that existing scaling laws do not account for.